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Chapter 1. The New C++17 Features 

In this chapter, we will cover thefollowing recipes: 

• Using structured bindings to unpack bundled return values 

• Limiting variable scopes to if and switch statements 

• Profiting from the new bracket initializer rules 

• Letting the constructor automatically deduce the resulting template class 
type 

• Simplifying compile-time decisions with constexpr-if 

• Enabling header-only libraries with inline variables 

• Implementing handy helper functions with fold expressions 



Introduction 


C++ got a lot of additions in C++11, C++14, and, most recently, C++17. By 
now, it is a completely different language compared to what it was just a 
decade ago. The C++ standard does not only standardize the language, as it 
needs to be understood by the compilers, but also the C++ standard template 
library (STL). 

This book explains how to put the STL to the best use with a broad range of 
examples. But at first, this chapter will concentrate on the most important 
new language features. Mastering them will greatly help you write A readable, 
maintainable, and expressive code a lot. 

We will see how to access individual members of pairs, tuples, and structures 

/\ 

comfortably with structured bindingsA and how to limit variable scopes with 
the newif and switch variable initialization capabilities. The syntactical 
ambiguities, which were introduced by C++11 with the new bracket 
initialization syntax, which looks the same for initializer lists, were fixed 
by new bracket initializer rules. The exact/ypeof template class instances can 
now bQdeducedfxom the actual constructor arguments, and if different 

/V 

specializations of a template class willA result in completely different code, 

/\ _ 

this is now easily expressibleA with constexpr-if. The handling of variadic 
parameter packs in template functions became much easier in many cases 
with the new/o/r/ expressions. At last, it became more comfortable to define 
static globally accessible objects in header-only libraries with the new ability 
to declare inline variables, which was only possible for functions before. 

Some of the examples in this chapter might be more interesting for 
implemented of libraries than for developers who implement applications. 
While we will have a look at such features for completeness reasons, it is not 
too critical to understand all the examples of this chapter immediately in 
order to understand the rest of this book. 




Using structured bindings toA unpack 
bundled return values 


1 


/\ 

C++17 comes with a new feature, whichA combines syntactic sugar and 
automatic type deduction: structured bindings. These help to assign values 
from pairs, tuples, and structs into individual variables. In other programming 
languages, this is also called unpacking. 

How to do it... 

Applying a structured binding in order to assign multiple variables from one 
bundled structure is always one step. Let's first see how it was done before 
C++17. Then, we can have a look at multiple examples that show how we 
can do it in C++17: 

/V 

• Accessing individual values of anA std: :pair: Imagine we have a 

/V 

mathematical function,A divide remainder, which accepts a dividend 

/v /\ 

and a divisor parameter and returns A the fraction of bothA as well as the 
remainder. It returns those values using an std: :pair bundle: 

std::pair<int, int> divide_remainder(int dividend, int di 


Consider the following way of accessing the individual values of the resulting 
pair: 


const auto result (divide_remainder(16, 3)); 
std::cout << "16 / 3 is " 

<< result.first << " with a remainder of " 

<< result.second << 1 \n'; 

Instead of doing it as shown in the preceding code snippet, we can now 

/V 

assign the individual valuesA to individual variables with expressive names, 

/V 

which is much better to A read: 

auto [fraction, remainder] = divide remainder(16, 3); 




std::cout << "16 / 3 is " 

<< fraction << " with a remainder of " 

<< remainder << '\n'; 

/v /v 

• Structured bindings also work withA std:: tupie:A Let's take the 
following example function, which gets us online stock information: 

std::tuple<std::string, 

std::chrono::system_clock::time_point, unsigne 
stock_info(const std::string Sname); 

Assigning its result to individual variables looks just like in the example 
before: 

const auto [name, valid_time, price] = stock_info("INTC") 

/v 

• Structured bindings also work with custom structures:A Let's assume a 
structure like the following: 

struct employee { 
unsigned id; 
std::string name; 
std::string role; 
unsigned salary; 

} ; 

Now, we can access these members using structured bindings. We can even 
do that in a loop, assuming we have a whole vector of those: 

int main() 

{ 

std::vector<employee> employees { 

/* Initialized from somewhere */}; 

for (const auto &[id, name, role, salary] : employees 
std::cout << "Name: " << name 

<< "Role: " << role 

<< "Salary: " << salary << '\n 1 ; 

} 

} 


How it works 



Structured bindings are always applied with the same pattern: 

auto [varl, var2, ...] = <pair, tuple, struct, or array expressio 

.-T— /V 

• The list of variablesA varl, var2 , ... must exactly match the number 
of variables contained by the expression being assigned from. 

• The <pair, tuple, struct, or array expression! must be One of 
the following: 

• An std: :pair. 

• An std::tuple. 

• A struct. All members must be non-static and defined in the same 
base class. The first declared member is assigned to the first 
variable, the second member to the second variable, and so on. 

• An array of fixed size. 

• The type can be auto, const auto, const autos, and even auto&s.A 


Note 


/V 

Not only for the sake of performance, always makeA sure to 
minimize needless copies by using references when appropriate. 

If we write too many or not enough variables between the square brackets, 
the compiler will error out, telling us about our mistake: 

std::tuple<int, float, long> tup {1, 2.0, 3}; 
auto [a, b] = tup; // Does not work 

This example obviously tries to stuff a tuple variable with three members into 
only two variables. The compiler immediately chokes on this and tells us 
about our mistake: 

error: type 'std::tuplecint, float, long>' decomposes into 3 elem 
auto [a, b] = tup; 

There’s more... 


A lot of fundamental data structures from the STL are immediately accessible 

/\ 

using structured bindings without us having to change anything.A Consider, 



for example, a loop that prints all the items of an std: :map: 

std::map<std::string, size_t> animal_population { 
{"humans", 7000000000}, 

{"chickens", 17863376000}, 

{"camels", 24246291}, 

{"sheep", 1086881528}, 

/* a! */ 


for 


} 


(const auto & [species, count] : animal_population) { 
std::cout << "There are " << count << " " << species 
<< " on this planet.\n"; 


This particular example works because when we iterate over anA std: :map 

/V /V 

Container, we get theA std: :pair<const key_type, value_type>A nodes 
on every iteration step. Exactly these nodes are unpacked using the structured 
bindings feature (key type is the species string and vaiue type is the 

/v 

population count A sizet) in order to access them individually in the loop 
body. 


Before C++17, it was possible to achieve a similar effect using std:: tie: 


int remainder; 

std::tie (std::ignore, remainder) = divide_remainder(16, 5); 
std::cout << "16 % 5 is " << remainder << 1 \n 1 ; 

This example shows how to unpack the resulting pair into two variables. 
TheA std:: tie is less powerful than structured bindings in the sense that we 
have to define all the variables we want to bind to before. On the other hand, 

/V 

this example shows a strength of std:: tieA that structured bindings do not 
have: the value std:: ignore acts as a dummy variable. The fraction part of 
the result is assigned to it, which leads to that value being dropped because 
we do not need it in that example. 


Note 


/V 

When using structured bindings, we don't have tieA dummy 

/V 

variables, so we have to bindA all the values to 



/V 

namedA variables. Doing so and ignoring some of them is 
efficient, nevertheless, because the compiler can optimize the 
unused bindings out easily. 

Back in the past, the divide remainder function could have been 
implemented in the following way, using output parameters: 


bool divide_remainder(int dividend, int divisor, 

int Sfraction, int Sremainder); 


Accessing it would have looked like the following: 


int fraction, remainder; 

const bool success {divide_remainder(16, 3, fraction, 
if (success) { 

std::cout << "16 / 3 is " << fraction << " with a 
<< remainder << '\n'; 


} 


remainder)} 

remainder o 


A lot of people will still prefer this over returning complex structures like 

/\ 

pairs, tuples, and structs, arguing that this way the code would beA faster, 
due to avoided intermediate copies of those values. This is not true any 
longer for modern compilers, which optimize intermediate copies away. 


Note 


Apart from the missing language features in C, returning complex 
structures via return value was considered slow for a long time 
because the object had to be initialized in the returning function 

/V /V 

and then copied into the variable thatA shouldA contain the 
return value on the caller side. Modern compilers support return 
value optimization (RVO), which enables for omitting 
intermediate copies. 



Limiting variable scopes to if and switch 
statements 


It is good style to limit the scope of variables as much as possible. 

Sometimes, however, one first needs to obtain some value, and only if it fits a 
certain condition, it can be processed further. 

/V /V 

For this purpose, C++ 17 comes with if A and switchA statements with 
initializers. 


How to do it... 


/V 

In this recipe, we use the initializer syntax inA both the supported contexts in 

/V 

order to see howA they tidy up our code: 

/V 

• The if statements:A Imagine we want to find a character in a character 
map using the find method of std: :map: 

if (auto itr (character_map. f ind(c) ); itr != charactermap 
// *itr is valid. Do something with it. 

} else { 

// itr is the end-iterator. Don't dereference. 

} 

// itr is not available here at all 

• The switch statements: This is how it would look to get a character 
from the input and, at the same time, check the value in a switch 
statement in order to control a computer game: 

switch (char c (getchar()); c) { 


case 

' a' 

move 

left 

) ; 

break; 

case 

' s ' 

move 

back 

) ; 

break; 

case 

'w' 

move 

f wd () 

r 

break; 

case 

'd' 

move 

right(); 

break; 

case 

' q' 

quit 

game 

) • 

/ r 

break; 




case 'O'...'9': select tool ( '0' - c); break; 


default: 

std::cout << "invalid input: " << c << '\n'; 


How it works... 

The if and switch statements with initializers are basically just syntax sugar. 
The following two samples are equivalent: 


Before A C++17: 

{ 

auto var (init value); 
if (condition) { 

// branch A. var is accessible 
} else { 

// branch B. var is accessible 

} 

// var is still accessible 

} 

Since C++17: 

if (auto var (init_value) ; condition) { 
// branch A. var is accessible 
} else { 

// branch B. var is accessible 

} 

// var is not accessible any longer 


The same applies to switch statements: 

Before C++17: 

{ 

auto var (init value); 
switch (var) { 
case 1: ... 

case 2: ... 


} 



// var is still accessible 

} 

Since C++17: 

switch (auto var (init_value); var) { 
case 1: ... 

case 2: ... 

} 

// var is not accessible any longer 


This feature is very useful to keep the scope of a variable as short as 
necessary. Before C++17, this was only possible using extra braces around 
the code, as the pre-C++17 examples show. The short lifetimes reduce the 

number of variables in the scope, which keeps our code tidy and makes it 

/\ 

easier to refactor.A 

There’s more... 


/V 

Another interesting use case is the limited scope ofA critical sections. 
Consider the following example: 


if (std::lock_guard<std::mutex> lg {my_mutex} ; somecondition) { 
// Do something 

} 


_ /v /v 

At first, an std:: lock guard is created. This is a classA that acceptsA a 
mutex argument as a constructor argument. It locks the mutex in its 

/V 

constructor, and when it runs out of scope, it unlocks A it again in its 
destructor. This way, it is impossible to forgetA to unlock the mutex. Before 
C++17, a pair of extra braces was needed in order to determine the scope 
where it unlocks again. 

/V 

Yet another interesting use case is the scope ofA weak pointers. Consider the 
following: 

if (auto shared_pointer (weak_pointer.lock ()); sharedpointer != 
// Yes, the shared object does still exist 
} else { 

// shared pointer var is accessible, but a null pointer 



} 

// shared_pointer is not accessible any longer 

This is another example where we would have a useless shared pointer 
variable leaking into the current scope, although it has a potentially useless 
state outside the if conditional block or noisy extra brackets! 

/V 

The i f A statements with initializers are especially useful when using legacy 
APIs with output parameters: 

if (DWORD exit_code; GetExitCodeProcess(process_handle, &exit_cod 
std::cout << "Exit code of process was: " << exit_code << '\n 

} 

// No useless exit code variable outside the if-conditional 


GetExitCodeProcess is a Windows kernel API function. It returns the exit 
code for a given process handle but only if that handle is valid. After leaving 
this conditional block, the variable is useless, so we don't need it in any scope 
any longer. 

Being able to initialize variables within if blocks is obviously very useful in 
a lot of situations and, especially, when dealing with legacy APIs thatA use 
output parameters. 


Note 


Keep your scopes tight using if and switch statement 
initializers. This makes your code more compact, easier to read, 
and in code refactoring sessions, it will be easier to move around. 



Profiting from the new bracket initializer 
rules 


C++11 came with the new brace initializer syntax {}. Its purpose was to 
allow for aggregate initialization, but also for usual constructor calling. 
Unfortunately, it was too easy to express the wrong thing when combining 
this syntax with the auto variable type. C++17 comes with an enhanced set of 
initializer rules. In this recipe, we will clarify how to correctly initialize 
variables with which syntax in C++17. 

How to do it... 


Variables are initialized in one step. Using the initializer syntax, there are two 
different situations: 

• Using the brace initializer syntax withoutauto type deduction: 

// Three identical ways to initialize an int: 
int xl = 1; 
int x2 {1} ; 

int x3 (1); 


std::vector<int> 

vl 

U, 

2, 

3}; 

// 

Vector with 

three ints 

std::vector<int> 

v2 = 

u. 

2, 

3}; 

// 

same here 


std::vector<int> 

v3 

(10, 

20) ; 

// 

// 

Vector with 10 ints, 
each have value 20 


• Using the brace initializer syntax with auto type deduction: 


auto v {1}; 
auto w {1, 2 } ; 

auto x = {1} ; 
auto y = {1, 2 } ; 
auto z = {1, 2, 3.0}; 


// v is int 

// error: only single elements in di 
// auto initialization allowed! (thi 
// x is std::initializer_list<int> 

// y is std::initializer_list<int> 

// error: Cannot deduce element type 




How it works... 


Without auto type deduction, there s not much to he surprised cibout in the 
brace {} operator, at least, when initializing regular types. When initializing 

/V 

containers such asA std:: vector, std:: list, and so on, a brace initializer 
will match the std:: initiaiizeriist constructor of that container class. It 
does this in a greedy manner, which means that it is not possible to match 

non-aggregate constructors (non-aggregate constructors are usual 

/\ 

constructors in contrast to the A ones that accept an initializer list). 

std:: vector, for example, provides a specific non-aggregate constructor, 

/V 

which fills arbitrarily many items with the same value:A std:: vector<int> 
v (N, value) . When writing std: :vector<int> v {N, value}, the 
initiaiizeriist constructor is chosen, which will initialize the vector with 

/v _ 

twoA items: n and value. This is a special pitfall one should know about. 

One nice detail about the {} operator compared to constructor calling using 

/\ 

normal () parentheses is A that they do no implicit type conversions: int x 

/V /\ 

(l . 2 ) ; A and int x = 1.2; will initialize x to value l A by silently rounding 
down the floating point value and converting it to int. int x {1.2};, in 
contrast, will not compile because it wants to exactly match the constructor 
type. 


Note 

One can controversially argue about which initialization style is 
the best one.Fans of the bracket initialization style say that using 
brackets makes it very explicit, that the variable is initialized with 
a constructor call, and that this code line is not 

/V 

reinitializing A anything. Furthermore, using {} brackets will 
select the only matching constructor, while initializer lines using 
() parentheses try to match the closest constructor and even do 
type conversion in order to match. 

The additional rule introduced in C++17 affects the initialization with auto 
type deduction—while C++11 would correctly make the type of the 







variableA auto X { 123 } ; A an std: : initializer_list<int> with only one 
element, this is seldom what we would want. C++17 would make the same 
variable an int. 

Rule of thumb: 


• auto var_name {one_element } ; deduces var_name to be of the same 
type as one_element 

• auto var_name {elementl, element2, ...}; is invalid and does not 
compile 

• auto var_name = {elementl, element2, ...}; deduces to an 
std:: initiaiizer_iist<T> with t being of the same type as all the 
elements in the list 

C++17 has made it harder to accidentally define an initializer list. 

Note 

Trying this out with different compilers in C++11/C++14 mode 
will show that some compilers actually deduce auto x { 123 }; to 
an int, while others deduce it to std: : initializer_list<int>. 
Writing code like this can lead to problems regarding portability! 



Letting the constructor automatically 
deduce the resulting template class type 


A lot of classes in C++ are usually specialized on types, which could be 
easily deduced from the variable types the user puts in their constructor calls. 
Nevertheless, before C++17, this was not a standardized feature. C++17 lets 
the compiler automatically deduce template types from constructor calls. 


How to do it... 


/V 

A very handy use case for this is constructingA std: :pair and 

/y _ /y 

std:: tupieA instances. These can be specialized and instantiatedA and 
specialized in one step: 


std::pair my_pair (123, "abc"); // std::pair<int, const c 

std::tuple my_tuple (123, 12.3, "abc"); // std::tuplecint, double 

// const char* 


How it works... 


/y /y 

Letas define anA example class whereA automatic template type deduction 
would be of value: 


template <typename Tl, typename T2, typename T3> 
class my_wrapper { 

Tl tl; 

T2 t2; 

T3 t3; 


public: 

explicit my_wrapper(Tl tl_, T2 t2_, T3 t3_) 
: tl{tl }, t2{t2 }, t3{t3 } 




/* a! */ 


}; 


Okay, this is just another template class. We previously had to write the 
following in orderA to instantiate it: 

my_wrapper<int, double, const char *> wrapper {123, 1.23, "abc"} 


We can now just omit the template specialization part: 

my_wrapper wrapper {123, 1.23, "abc"}; 

Before C++17, this was only possible by implementing a make function 
helper. 

my_wrapper<Tl, T2, T3> make_wrapper(T1 tl, T2 t2, T3 t3) 

{ 

return {tl, t2, 13} ; 

} 

Using such helpers, it was possible to have a similar effect: 

auto wrapper (make_wrapper(123, 1.23, "abc")); 

Note 

The STL already comes with a lot of helper functions such 

/\ 

asA that one: std: :make_shared, std: :make_unique, 
std: :make_tupie, and so on. In C++17, these can now mostly be 
regarded as obsolete. Of course, they will be provided further for 
compatibility reasons. 


There’s more... 

/V 

WhatA we just learned about was implicit template type deduction. In some 
cases, we cannot rely on implicit type deduction. Consider the following 



example class: 


template <typename T> 
struct sum { 

T value; 

template <typename ... Ts> 

sum(Ts&& ... values) : value{(values + ...)} {} 

}; 


, /v /v 

This struct, A sum, accepts an arbitrary number of parameters and adds A them 
together using a fold expression (have a look at the fold expression recipe a 
little later in this chapter to get more details on fold expressions). The 

/V 

resulting sum is saved in the member variableA value. Now the question is, 
what type is t? If we don't want to specify it explicitly, it surely needs to 
depend on the types of the values provided in the constructor. If we provide 
string instances, it needs to be std: : string. If we provide integers, it needs 
to be int. If we provide integers, floats, and doubles, the compiler needs to 
figure out which type fits all the values without information loss. In order to 
achieve that, we provide an explicit deduction guide: 

template ctypename ... Ts> 

sum(Ts&& ... ts) -> surrKstd: : common_type_t<Ts . . . »; 

This deduction guide tells the compiler to use the std: : common type t trait, 
which is able to find out which common type fits all the values. Let's see 

/V 

howA to use it: 

sum s {lu, 2.0, 3, 4.Of}; 

sum string_sum {std::string{"abc"}, "def"}; 

std::cout << s.value << '\n' 

<< string_sum.value << '\n'; 

In the first line we instantiate a sum object with constructor arguments of type 

/\ 

unsigned, double, int, and float. TheA std: : common_type_t returns 
double as the common type, so we get a sum<doubie> instance. In the second 
line, we provide an std: : string instance and a C-style string. Following our 
deduction guide, the compiler constructs an instance of type 

surrKstd: :string>. 



When running this code, it will print 10 as the numeric sum and abodef as the 
string sum. 



Simplifying compile time decisions with 
constexpr-if 


In templated code, it is often necessary to do certain things differently, 
depending on the type the template is specialized for. C++17 comes with 
constexpr-if expressions, which simplify the code in such situations a lot. 


How to do it... 


In this recipe, we'll implement a little helper template class. It can deal with 
different template type specializations because it is able to select completely 
different code in some passages, depending on what type we specialize it for: 

/V 

1. Write the part of the code that A is generic. In our example, it is a simple 
class, which supports adding a type u value to the type t member value 

/V 

using an A add function: 

template ctypename T> 
class addable 
{ 

T val; 
public: 

addable (T v) : val{v} {} 

template ctypename U> 

T add(U x) const { 
return val + x; 

} 

}; 


/V /\ 

2. Imagine that type t isA std:: vector<something>A and type u is just 
int. What shall it mean to add an integer to a whole vector? Let's say it 
means that we add the integer to every item in the vector. This willA be 




done in a loop: 


template <typename U> 

T add(U x) 

{ 

auto copy (val); // Get a copy of the vector member 
for (auto &n : copy) { 
n += x; 

} 

return copy; 

} 

/v _ 

3. A The next and last step is to combine both worlds. If t is a vector of u 
items, do the loop variant. If it is not, just implement the normal 

/V 

addition: A 

template <typename U> 

T add(U x) const { 

if constexpr (std;;is_same_v<T, std::vector<U») { 

auto copy (val); 
for (auto &n : copy) { 
n += x ; 

} 

return copy; 

} else { 

return val + x; 

} 

} 

4. The class can now be put to use. Let's see how nicely it works with 
completely different types, such as int, float, std:: vector<int>, and 
std::vector<string>: 

addable<int>{1}.add(2); // is 3 

addable<float>{1.0}.add(2) ; // is 3.0 

addable<std::string>{"aa"}.add("bb"); // is "aabb" 

std::vector<int> v {1, 2, 3}; 

addable<std: :vector<int»(v} .add(10) ; 

// is std::vector<int>{11, 12, 13} 

std::vector<std::string> sv {"a", "b", "c"}; 

addable<std::vector<std::string>>{sv}.add(std::string}"z"} 
// is }"az", "bz", "cz"} 



How it works... 


The new constexpr-if works exactly like usualA if-else constructs. The 

/\ 

difference is that the condition thatA it tests has to be evaluated at compile 

/V 

time. All runtime code thatA the compiler creates from our program will not 
contain any branch instructions from constexpr-if conditionals. One could 

/V /V /V 

also put it that it works in a similar manner to A preprocessorA #if A and 
#eise text substitution macros, but for those, the code would not even have to 

/V 

be syntactically well-formed. A All the branches of a constexpr-if construct 

/V 

need to be syntactically well-formed, but the branches thatA are not taken do 
not need to be semantically valid. 

/\ /v /v /v 

In order to distinguish whetherA the code shouldA add the valueA xA to a 

/V 

vector or not, we use the type traitA std:: is same. An 

/V /V 

expressionA std:: is_same<A, b> :: valueA evaluates to the Boolean 

/V /\ 

valueA true A if a and b are of the same type. The condition used in our 

recipe is std: :is_same<T, std: :vector<U>>: : value, which evaluates to 
/\ /\ 

trueA if the user specialized the class on t =a std:: vector<x>A and tries 

/\ 

to call add with a parameter of type u =A x.A 

There can, of course, be multiple conditions in one constexpr-if-else block 
(note that a and b have to depend on template parameters and not only on 
compile-time constants): 

if constexpr (a) { 

// do something 
} else ifconstexpr (b) { 

// do something else 
} else { 

// do something completely different 

} 

With C++17, a lot of meta programming situations are much easier to express 
and to read. 


There’s more... 


In order to relate how much constexpr-if constructs are an improvement to 



C++, we can have a look at how the same thing could have been implemented 
before C++17: 


template <typename T> 
class addable 
{ 

T val; 
public: 

addable(T v) : val{v} {} 
template <typename U> 

std::enable_if_t<!std::is_same<T, std::vector<U»::value, T> 

add(U x) const { return val + x; } 

template <typename U> 

std::enable_if_t<std::is_same<T, std::vector<U»::value, 

std: :vector<U» 

add(U x) const { 

auto copy (val); 
for (auto &n : copy) { 
n += x ; 

} 

return copy; 

} 

}; 


Without using constexpr-if, this class works for all different types we wished 
for, but it looks super complicated. How does it work? 

/v 

The implementations alone of the two different A add functions look simple. 
It's their return type declaration, which makes them look complicated, and 
which contains a trick—an expression such 

/\ f\ 

asA std: : enable_if_t<condition, type> evaluates to typeA if condition 

/\ /v 

is true. Otherwise, theA std: :enabie_if_tA expression does not evaluate 
to anything. That would normally considered an error, but we will see why it 
is not. 

/V /\ 

For the second add function, the sameA condition is used in anA inverted 
manner. This way, it can only be true at the same time for one of the 

/V 

two A implementations. 

When the compiler sees different template functions with the same name and 



has to choose one of them, an important principle comes into play: SFINAE, 
which stands for Substitution Failure is not an Error. In this case, this 
means that the compiler does not error out if the return value of one of those 
functions cannot be deduced from an erroneous template expression (which 
std: :enabie_if is, in case its condition evaluates to false). It will simply 
look further and try the other function implementation. That is the trick; that 
is how this works. 

/\ 

What a hassle.A It is nice to see that this became so much easier with C++17. 



Enabling header-only libraries with inline 
variables 


While it was always possible in C++ to declare individual functions inline , 

/\ 

C++17 additionallyA allows us to declare variables inline. This makes it 
much easier to implement header-only libraries, which was previously only 
possible using workarounds. 


How it’s done... 


/V 

In this recipe, we create an exampleA class that could suit as a member of a 
typical header-only library. The target is to give it a static member and 
instantiate it in a globally available manner using the inline keyword, which 
would not be possible like this before C++17: 

/V /V 

1. TheA process monitor class shouldA both contain a static member and 
be globally accessible itself, which would produce double-defined 
symbols when included from multiple translation units: 


// foo_lib.hpp 

class process_monitor { 
public: 

static const std::string standard_string 

{"some static globally available string"}; 

}; 


process_monitor global_process_monitor; 

2. If we now include this in multiple . cpp files in order to compile and link 
them, this would fail at the linker stage. In order to fix this, we add the 
inline keyword: 


// foo_lib.hpp 


class process monitor { 




public: 

static const inline std::string standard_string 
{"some static globally available string"}; 

}; 


inline process_monitor global_process_monitor; 


Voila, that's it! 

How it works... 


C++ programs do often consist of multiple C++ source files (these do 
have, cppor.ccsuffiees). These are individually compiled to modules/object 
files (which usually have .o suffices). Linking all the modules/object files 
together into a single executable or shared/static library is then the last step. 

At the link stage, it is considered an error if the linker can find the definition 
of one specific symbol multiple times. Let's say, for example, we have a 

/V 

function with a signature such asA int f oo ();. If two modules define the 
same function, which is the right one? The l ink er can't just roll the dice. Well, 
it could, but that's most likely not what any programmer would ever want to 
happen. 

The traditional way to provide globally available functions is todeclarethem 

/V 

in the header files, which will be included by any C++ module thatA needs to 
call them. The definition of every of those functions will be then put once 

into separate A module files. These are then linked together with the modules 

/\ /\ _ 

thatA desireA to use these functions. This is also called the One Definition 
Rule (ODR). Check out the following illustration for better understanding: 



.CPP 


.CPP 


.CPP 



/\ /V 

However, if this wereA the only way, then it would notA have been possible 
to provide header-only libraries. Header-only libraries are very handy 
because they only need to be included using # include into any C++ program 

/V 

file and then are immediately available. In order to use libraries that A are not 
header-only, the programmer must also adapt the build scripts in order to 
have the linker link the library modules together with his own module files. 
Especially for libraries with only very short functions, this is unnecessarily 
uncomfortable. 

/\ 

For such cases, theiniinekeyword can be used to make an exceptionA in 
order to allow multiple definitions of the same symbol in different modules. 
If the linker finds multiple symbols with the same signature, but they are 










declared inline, it will just choose the first one and trust that the other 
symbols have the same definition. That all equal inline symbols are defined 
completely equal is basically a promise from the programmer. 

Regarding our recipe example, the linker will find the 

process_monitor : : standard_string Symbol in every module 

/v /\ 

thatA includes foo_iib.hpp.A Without the inline keyword, it would not 
know which one to choose, so it would abort and report an error. The same 

/V 

applies to the giobai process monitorA symbol. Which one is the right 
one? 

After declaring both the symbols inline, it will just accept the first 

/V 

occurrenceA of each symbol and drop all the others. 

Before C++17, the only clean way would be to provide this symbol via an 

/V 

additionalA C++ module file, which would force our library users to include 
this file in the linking step. 

/\ /V 

The inline keyword traditionally also hasA another function.A It tells the 
compiler that it can eliminate the function call by taking its implementation 
and directly putting it where it was called. This way, the calling code contains 
one function call less, which can often be considered faster. If the function 

/V 

isA very short, the resulting assembly will also be shorter (assuming that the 

/\ 

number of instructions thatA do the function call, saving and restoring the 
stack, and so on, is higher than the actual payload code). If the inlined 
function is very long, the binary size will grow and this might sometimes not 

A _ 

even lead toA faster code in the end. Therefore, the compiler will only use the 
inline keyword as a hint and might eliminate function calls by inlining 
them. But it can also inline some functions without the programmer having it 
declared inline. 


There’s more... 

One possible workaround before C++17 was providing astatic function, 
which returns a reference toastatic object: 


class foo { 
public: 



static std::string& standard_string() { 

static std::string s {"some standard string"}; 
return s; 

} 

}; 


This way, it is completely legal to include the header file in multiple modules 
but still getting access to exactly the same instance everywhere. However, the 
object is/io/constructed immediately at the start of program but only on the 
first call of this getter function. For some use cases, this is indeed a problem. 
Imagine that we want the constructor of the static, globally available object to 
do something important at program start (just as our reciple example library 
class), but due to the getter being called near the end of the program, it is too 
late. 

/V /V 

Another workaround is to make the non-template classA fooA a template 
class, so it can profit from the same rules as templates. 

Both strategies can be avoided in C++17. 



Implementing handy helper functions with 
fold expressions 


Since C++11, there are variadic template parameter packs, which enable 

/V 

implementing functions that acceptA arbitrarily many parameters. 

Sometimes, these parameters are all combined into one expression in order to 
derive the function result from that. This task became really easy with 
C++17, as it comes with fold expressions. 

How to do it... 


Let's implement a function that takes arbitrarily many parameters and returns 
their sum: 

1. At first, we define its signature: 

template <typename ... Ts> 
auto sum(Ts ... ts); 

/V /V 

2. So, we have a parameter packA tsA now, and the function 

/V 

shouldA expand all the parameters and sum them together using a fold 
expression. If we use any operator (+, in this example) together with 

/V 

... A in order to apply it to all the values of a parameter pack, we need 
to surround the expression with parentheses: 

template <typename ... Ts> 
auto sum(Ts ... ts) 

{ 

return (ts + . . . ); 

} 

3. We can now call it this way: 

int the sum (sum(l, 2, 3, 4, 5)}; // Value: 15 





/\ 

4. It does not only work with int types; we can call it with anyA type 

/\ /\ 

thatA just implements the + operator, such asA std:: string: 

std::string a {"Hello "}; 
std::string b {"World"}; 

std::cout << sum(a, b) << '\n'; // Output: Hello World 


How it works... 

/V 

What we just did was a simpleA recursive application of a binary operator (+) 
to its parameters. This is generally called folding. C++17 comes with fold 
expressions, which help expressingthe same ideawith less code. 

This kind of expression is called unary fold. C++17 supports folding 
parameter packs with the following binary operators: +, -, *, /, %, A , &, |, =, <, 

>, «, >>, +=, -=, *=, /=, %=, A =, &=, | =, <<=, >>=,==, ! =, <=, >=, &&, | | - 

>*. 

By the way, in our example code, it does not matter if we write (ts + 

/\ /\ 

a |) A or (a | + ts); both work. However,A there is a difference that may be 

/V 

relevant in other cases—if the a | dots are on the right-handA side of the 
operator, the fold is called a right fold. If they are on the left-handA side, it is 
a left fold. 

In our sum example, a unary left fold expands toi+ (2+ (3+ (4 + 5))), 

/V 

while a unary right fold willA expand to (((l + 2) + 3) + 4) + 5. 
Depending on the operator in use, this can make a difference. When adding 
numbers, it does not. 

There’s more... 

/V 

In case someone calls sum () with no A arguments, the variadic parameter 
pack contains no values that could be folded. For most operators, this is an 

/V 

error (for some, it is not;A we will see this in a minute). We then need to 

/V /\ 

decideA if this shouldA stay an error or if an empty sum should result in a 
specific value. The obvious idea is that the sum of nothing is o. 



This is how itas done: 


template <typename ... Ts> 
auto sum(Ts ... ts) 

{ 

return (ts + ... + 0 ) ; 

} 

This way, sum ( ) evaluates to o, and sum (i, 2 , 3) evaluates to (l + (2 + 
(3 + 0 ))) . A Such folds with an initial value are called binary folds. 

Again, it works if we write (ts + ... + o),or (0 + ... + ts), but this 
makes the binary fold a binary right fold or a binary left fold again. Check 
out the following diagram: 



When using binary folds in order to implement the no-argument case, the 
notion of an identity element is often important—in this case, adding a 0 to 
any number changes nothing, which makes 0 an identity element. Because of 
this property, we can add a 0 to any fold expression with the operators + or -, 
which leads to the result 0 in case there are no parameters in the parameter 




pack. From a mathematical point of view, this is correct. From an 
implementation view, we need to define what is correct, depending on what 
we need. 

The same principle applies to multiplication. Here, the identity element is 1 : 

template <typename ... Ts> 
auto product(Ts ... ts) 

{ 

return (ts * ... * 1); 

} 

The result of product (2, 3 ) is 6, and the result of product () without 
parameters is l. 

The logical and (&&) and or (| |) operators come with built-in identity 

/V 

elements. Folding an empty parameter pack withA && results in true, and 
folding an empty parameter pack withA | | results in false. 

/\ 

Another operator thatA defaults to a certain expression when applied on 
empty parameter packs is the comma operator (,), which then defaults to 

void () . 


In order to ignite some inspiration, let's have a look at some more little 

/V 

helpers thatA we can implement using this feature. 


Match ranges against individual items 

/V /V 

How about a function thatA tells whetherA some range contains at least one 
of the values we provide as variadic parameters: 


template <typename R, typename ... Ts> 
auto matches(const R& range, Ts ... ts) 

{ 

return (std::count(std::begin(range), std::end(range) , ts) + 

} 

The helper function uses the std:: count function from the STL. This 
function takes three parameters: the first two parameters are the begin and 

/V 

end iterators of some iterable range, and as theA third parameter, it takes a 



value which will be compared to all the items of the range. 

/V /V 

TheA std:: count A method then returns the number of all the elements 

/V 

within the range thatA are equal to the third parameter. 

In our fold expression, we always feed the begin and end iterators of the same 

/V 

parameter range into the std:: count function. However,A as the third 
parameter, each time we put one other parameter from the parameter pack 
into it. In the end, the function sums up all the results and returns it to the 
caller. 

We can use it like this: 


std::vector<int> v {1, 2, 3, 4, 5}; 


matches 

(v. 

2, 5) 

1 ; 

// 

returns 

2 

matches 

(v. 

100, 

200) ; 

// 

returns 

0 

matches 

("abcdefg", 

’x’. 

1 y 1 , 'z' 

); // 

returns 

0 

matches 

("abcdefg", 

’ a ’ , 

■d’, ’f’ 

); // 

returns 

3 


As we can see, the matches helper function is quite versatile—it can be called 
on vectors or even on strings directly. It would also work on initializer lists, 
On instances of std: : list, std: : array, std: : set, and SO On! 


Check if multiple insertions into a set are successful 


/V 

Let's write a helper thatA inserts an arbitrary number of variadic parameters 
into an std:: set and returns if all the insertions are successful : 

template <typename T, typename ... Ts> 
bool insert_all(T &set, Ts ... ts) 

{ 

return (set.insert(ts).second && ...); 

} 

So, how does this work? The insert function of std:: set has the following 
signature: 


std::pairtiterator, bool> insert(const value_type& value); 

The documentation says that when we try to insert an item, the insert 



function will return an iterator and a bool variable in a pair. The bool 

/V 

value is true A if the insertion is successful. If it is successful, the iterator 
points to the new element in the set. Otherwise, the iterator points to the 
existing item, which would collide with the item to be inserted. 

/V 

Our helper function accesses theA . second field after insertion, which is just 

/V 

the bool variable thatA reflects success or fail. If all the insertions lead to 

true in all the return pairs, then all the insertions were successful. The fold 

/\ 

expression combines all the insertion results with the &&A operator and 
returns the result. 

We can use it like this: 


std::set<int> my_set {1, 2, 3}; 

insert_all(my_set, 4, 5, 6); // Returns true 

insert_all(my_set, 7, 8, 2); // Returns false, because the 2 coll 

/v 

Note that if we try to insert, for example, three elements, butA the second 
element can already not be inserted, the & & ... fold will short-circuit and 
stop inserting all the other elements: 

std::set<int> my_set {1, 2, 3}; 

insert_all(my_set, 4, 2, 5); // Returns false 
// set contains {1, 2, 3, 4} now, without the 5! 


Check if all the parameters are within a certain range 

If we can check if one variable is within some specific range, we can also do 
the same thing with multiple variables using fold expressions: 

template <typename T, typename ... Ts> 
bool within (T min, T max, Ts ...ts) 

{ 

return ((min <= ts && ts <= max) && ...); 

} 


The expression, (min <= ts && ts <= max) , does tell for every value of the 



parameter pack if it is between min and max (includingmin and max). We 

/V 

choose the &&A operator to reduce all the Boolean results to a single one, 

/V 

which is only true A if all the individual results are true. 

This is how it looks in action: 

within ( 10, 20, 1, 15, 30); // --> false 

within( 10, 20, 11, 12, 13); // --> true 

within(5.0, 5.5, 5.1, 5.2, 5.3) // --> true 

Interestingly, this function is very versatile because the only requirement it 
imposes on the types we use is that they are comparable with the <= operator. 
And this requirement is also fulfilled by std:: string, for example: 

std::string aaa {"aaa"}; 
std::string bed {"bed"}; 
std::string def {"def"}; 
std::string zzz {"zzz"}; 

within (aaa, zzz, bed, def); // --> true 

within(aaa, def, bed, zzz); // --> false 


Pushing multiple items into a vector 

/\ 

It's also possible to write a helper thatA does not reduce any results but 
processes multiple actions of the same kind. Like inserting items into an 
std: : vector, which does not return any results (std: : vector : : insert () 

/V 

signalizes error byA throwing exceptions): 

template ctypename T, typename ... Ts> 

void insert_all(std::vector<T> &vec, Ts ... ts) 

{ 

(vec.push_back (ts), ...); 

} 

int main() 

{ 

std::vector<int> v {1, 2, 3}; 
insert_all(v, 4, 5, 6); 

} 



Note that we use the comma (,) operator in order to expand the parameter 

/\ /V 

pack into individual vec .push back (...) A calls without folding theA actual 
result. This function also works nicely with an empty parameter pack because 

/y /y 

the commaA operator has an implicit identity element,A void (), which 

/\ 

translates to do nothing .A 



Chapter 2. STL Containers 

We will cover the following recipes in this chapter: 

• Using the erase-remove idiom on std: : vector 

• Deleting items from an unsorted std:: vector in 0(1) time 

• Accessing std:: vector instances the fast or the safe way 

• Keeping std:: vector instances sorted 

• Inserting items efficiently and conditionally into std: :map 

• Knowing the new insertion hint semantics of std: :map: : insert 

• Efficiently modifying the keys of std: :map items 

• Using std: :unordered map with Custom types 

• Filtering duplicates from user input and printing them in alphabetical 
order with std: : set 

• Implementing a simple RPN calculator with std:: stack 

• Implementing a word frequency counter with std:: map 

• Implementing a writing style helper tool for finding very long sentences 
in texts with std: : set 

• Implementing a personal to-do list using std: :priority_queue 




Introduction 


The C++ standard library comes with a wide range of standard containers. A 
container always contains a collection of data or objects. The cool thing is 
that containers can be used with practically any kind of object, so we just 
need to pick the right containers for our specific application. The STL gives 
us stacks, automatically growing vectors, maps, and so on. This way we can 
concentrate on our app and don't need to reinvent the wheel. Knowing all 
containers well is therefore crucial for every C++ programmer. 

All containers the STL provides can be categorized as follows, which is 
explained in detail in the subsequent subsection: 

• Contiguous storage 

• List storage 

• Search trees 

• Hash tables 

• Container adapters 

Contiguous storage 

The simplest way to store objects is to put them just next to each other in one 
large chunk of memory. Such memory can be accessed in a random access 
manner in 0(1) time. 

The easiest way to do that is using std:: array, which is just a wrapper 
around normal C-style arrays. It should always be preferred over normal C- 

style arrays, because it comes with no runtime cost, but adds some comfort 

/\ 

and safety. Just as C-style arrays, it has a fixed size once it isA created. 

The std:: vector comes into play when array-like storage is needed, but with 
varying sizes. It uses memory from the heap to store objects. Whenever new 
objects are pushed into the vector exceeding its current size, it will 
automatically move all items to a larger chunk of newly allocated memory, 




and delete the old chunk. Furthermore, if a new item shall be inserted 

/V 

between old ones, it can even move the existing items backA and forth. If an 
item somewhere in the middle shall be removed, the vector class will 
automatically close the gap by moving the other objects together. 

If lots of objects are inserted/removed at the front and/back of an 

/V /\ 

std:: vector A that leads to a lot of newA memory allocations in order to 

/V 

gain space, with potentially costly object move,A std:: deque offers an 
interesting trade off here. The objects are stored in fixed-size chunks of 
contiguous memory, but these chunks are independent of each other. This 
makes it very simple and quick to arbitrarily grow the deque, because objects 
in existing chunks can stay where they are, whenever a new chunk is 
allocated and put at the front or the end of the container. Deque stands for 
double-ended queue. 


List storage 

The std:: list is a classical doubly-linked list. Not less, and not more. If 
only uni-directional list traversal is needed, std: : forward iist may be more 
performant in both space and maintenance complexity, because it maintains 
only list item pointers in one direction. Lists can only be traversed linearly 
with O(n) time. Inserting and removing items at specific positions can be 
done in 0(1) time. 

Search trees 

Whenever objects have a natural order so that they can be sorted using some 
notion of the mathematical smaller< relation, they can be maintained in that 
order using search trees. As the name suggests, search trees can easily be 
searched for specific items using a search key, which allows 0(log(n)) search 
times. 

The STL provides such trees in different flavors, where std:: set is the 
simplest one of them, storing just unique , sortable objects in a tree structure. 


std:: map is different in that regard, that it stores data in pairs. A pair consists 



of a key, and a value. The search tree uses the key part for sorting the items, 
which enables for using std: :map as an associative container. As in 
std:: set, all key items must only exist once in the whole tree. 

std::multiset and std::multimap are specializations, which drop the 

/v 

requirement forA uniqueness of the key objects. 

Hash tables 

When talking about associative containers, search trees are not the only way 
to implement them. With hash tables, items can be found in 0(1) time, but 
they ignore the natural order of the items, so they can't be easily traversed in 
a sorted manner. The size of the hash table can be manipulated by the user, 

and the hash function can also be chosen individually, which is important, 

/\ 

because the performance versusA space consumption characteristics depend 
on that. 


std: :unordered set and std: :unordered map have SO much interface in 

/v 

common with their std:: set and std: :mapA pendants, that they can be used 
almost interchangeably. 

Just as for the search tree implementations, both containers have their 
multipendants; std: :unordered_multiset and std: :unordered_multimap 

both drop the constraint on the uniqueness of the objects/keys, so we can 
store multiple elements with the same key. 

Container adapters 

Arrays, lists, trees, and hash tables are not the only way to store and access 

/V 

data, as there are also stacks, queues, and so on. Similar,A more sophisticated 
structures, however, can be implemented using the more primitive ones, and 
the STL does that with the following ones in the form of container adapters : 

std: : stack, std: : queue, and std: :priority_queue. 

The cool thing is that whenever we need such a data structure, we can just 
pick such an adapter. Then, when we realize that they do not work out well 



regarding their performance, we can just change a template parameter in 
order to let the adapter use a different container implementation, and that's it. 

In practice, this means, for example, that we can switch the implementation 

/\ /\ 

of anA std: : stackA instance from std: : vector to std: : deque. 



Using the erase-remove idiom on std::vector 


A lot of novice C++ programmers learn about std:: vector, that it basically 
works like an automatically growing array, and stop right there. Later, they 

only lookup its documentation in order to see how to do very specific things, 

/\ 

for example,A removing items. Using STL containers like this will only 
scratch the surface of how much they help writing clean, maintainable, and 
fast code. 

^ /V 

This sectionA is all about removing items from in-between a vector 

/V 

instance.A When an item disappears from a vector, and sits somewhere in the 
middle between other items, then all items right from it must move one slot to 

/V 

the left (which gives A this task a runtime cost within 0(n)). Many novice 
programmers will do that using a loop, since it is also not really a hard thing 

/v /\ 

to do. Unfortunately, they will potentiallyA ignoreA a lot of optimization 
potential while doing that. And in the end, a hand crafted loop is neither 
faster, nor prettier to read than the STL way, which we will see next. 

How to do it... 

In this section, we are filling an std:: vector instance with some example 
integers, and then prune some specific items away from it. The way we are 
doing it is considered the correct way of removing multiple items from a 
vector. 

1. Of course we need to include some headers before we do anything. 

#include <iostream> 

#include <vector> 

#include <algorithm> 

2. Then we declare that we are using namespace std to spare us some 
typing. 


using namespace std; 



3. Now we create us a vector of integers and fill it with some example 
items. 

int main() 

{ 

vector<int> v {1, 2, 3, 2, 5, 2, 6, 2, 4, 8}; 

4. The next step is to remove the items. What do we remove? There are 
multiple 2 values. Let's remove them. 

const auto new_end (remove(begin(v), end(v), 2)); 

5. Interestingly, that was only one of the two steps. The vector still has the 
same size. The next line makes it actually shorter. 

v.erase(new_end, end(v)); 

6. Let's stop by here in order to print the vector's content to the terminal, 
and then continue. 

for (auto i : v) { 
cout << i << ", 

} 

cout << 1 \n 1 ; 

7. Now, let's remove a whole class of items, instead of specific values. In 
order to do that, we define a predicate function first, which accepts a 
number as parameter, and returns true, if it is an odd number. 

const auto odd ([](int i) { return i % 2 != 0; }); 

8. Now we use the remove if function and feed it with the predicate 
function. Instead of removing in two steps as we did before, we do it in 
one. 

v.erase(remove_if(begin(v), end(v), odd), end(v)); 

9. All odd items are gone now, but the vector's capacity is still at the old 10 

/\ 

elements.A In a last step, we reduce that also to the actual current size 
of the vector. Note that this might lead the vector code to allocate a new 
chunk of memory that fits and moves all items from the old chunk to the 



new one. 


v.shrink_to_fit(); 

10. Now, let's print the content after the second run of removing items and 
that's it. 

for (auto i : v) { 
cout << i << ", 

} 

cout << 1 \n 1 ; 

} 

11. Compiling and running the program yields the following two output 
lines from the two item removing approaches. 

$ ./main 

1, 3, 5, 6, 4, 8, 

6, 4, 8, 


How it works... 


What became obvious in the recipe is that when removing items from the 

/V 

middle of a vector, they first need to be removed A and then erased. At least 

/\ _ 

the functions we used have names A like this. This is admittedly confusing, 
but let's have a closer look at it to make sense of these steps. 

The code which removes all values of 2 from the vector, looked like this: 


const auto new_end (remove(begin(v), end(v), 2)); 
v.erase(new end, end(v)); 


The std:: begin and std: :end functions both accept a vector instance as 
parameter, and return us iterators, which point to the first item, and past the 

/V 

last item, just as sketchedA in the upcoming diagram. 

After feeding these and the value 2 to the std:: remove function, it will move 
the non -2 values forward, just like we could do that with a manually 



programmed loop. The algorithm will strictly preserve the order of all non-2 
values while doing that. A quick look at the illustration might be a bit 

confusing. In step 2 , there still is a value of 2, and the vector should have 

/\ 

become shorter, as there were fourA values of 2, which all ought to be 
removed. Instead, the 4 and the 8 which were in the initial array, are 
duplicated. What's that? 


1.) initial state 


begin end 


1 

2 

3 

2 

5 

2 

6 

2 

4 

8 


A 


2.) after new_end = std::remove ( begin, end, 2 ); 


n 

3 

5 

6 

D 

8 

6 

2 

D 

8 


/V A A 


begin new_end end 


3.) after vector.erase ( new end, end ); 


1 

3 

5 

6 

4 

8 


A A 


begin end 


/V 

Let's only takeA a look at all the items, which are within the range and which 

/\ 

spans from the begin iterator on the illustration, toA the new end iterator. 














The item, to which the new end iterator points, is the first item past the range, 
so it's not included. Just concentrating on that region (these are only the items 

/\ /\ /V 

fromA lA to includingA 8), we realize that this is the correct range from 
which all values of 2 are removed. 

This is where the erase call comes into play: We must tell the vector that it 
shall not consider all items from new end to end to be items of the vector any 
longer. This order is easy to follow for the vector, as it can just point its end 
iterator to the position of new end and it's done. Note that new end was the 
return value of the std:: remove call, so we can just use that. 

Note 

Note that the vector did more magic than just moving an internal 
pointer. If that vector was a vector of more complicated objects, it 
would have called all the destructors of the to-be-removed items. 

Afterward, the vector looks like in step 3 of the diagram: it's considered 
smaller now. The old items which are now out of the range, are still in 
memory. 

In order to make the vector occupy only as much memory as it needs, we 

/V 

make theA shrink to f it call in the end. During that call, it allocates 
exactly as much memory as needed, moves over all the items and deletes the 
larger chunk we don't need any longer. 

In step 8, we define a predicate function and use it with std:: remove! f in 
only one step. This works, because whatever iterator the remove function 
returns, it is safe to be used in the vector's erase function. Even if no odd item 
was found, the std:: remove if function will do just nothing , and return the 
end iterator. Then, a call like v. erase (end, end) ; also does nothing, hence 
it is harmless. 

There’s more... 


The std:: remove function also works on other containers. When used with 



/\ 

std:: array, A note that it does not support the second step of calling erase, 

because they do not have automatic size handling. Just because std:: remove 

/\ /\ 

effectively only moves items aroundA and does not performA their actual 

/V 

deletion, it can also be used on data structures such asA arrays that do not 
support resizing. In the array case, one could overwrite the values past the 

/V 

new end iterator with sentinel values such asA ' \o' for strings, for example. 



Deleting items from an unsorted std::vector 
in 0(1) time 


Deleting items from somewhere in the middle of an std:: vector takes O(n) 
time. This is because the resulting gap from removing an item must be filled 
by moving all the items which come after the gap one slot to the left. 

While moving items around like this, which might be expensive if they are 
complex and/or very large and include many items, we preserve their order. If 

preserving the order is not important, we can optimize this, as this 

/\ 

sectionA shows. 


How to do it... 

/\ 

In this section, we willA fill an std:: vector instance with some example 
numbers, and implement a quick remove function, which removes any item 
from a vector in 0(1) time. 

/V 

1. First, we need to include the requiredA header files. 

#include <iostream> 

#include <vector> 

#include <algorithm> 

2. Then, we define a main function where we instantiate a vector with 
example numbers. 

int main() 

{ 

std::vector<int> v {123, 456, 789, 100, 200}; 

3. The next step is to delete the value at index 2 (counting from zero of 

_ /\ 

course, so it's the third number 7 8 9). The function we willA use for that 
task is not implemented yet. We do that some steps later. Afterward, we 
print the vector's content. 




quick_remove_at(v, 2); 

for (int i : v) { 

std::cout << i << ", 

} 

std::cout << '\n'; 

/v 

4 . Now, we will delete another item. It willA be the value 123, and let's say 
we don't know its index. Therefore, we will use A the 

/V 

std:: findA function, which accepts a range (the vector), and a value, 

and then searches for the value's position. Afterward, it returns us an 

/\ 

iterator pointing to the 123 value. We will useA the same 

quick remove at function, but this is an overloaded version of the 

previous one which accepts iterators. It is also not implemented, yet. 

quick_remove_at(v, std::find(std::beqin(v), std::end(v) 

for (int i : v) { 

std::cout << i << ", 

} 

std::cout << '\n'; 

} 

5 . Apart from the two quick remove at functions, we are done. So let's 
implement these. (Note that they should be at least declared before the 
main function. So let's just define them there.) Both the functions accept 
a reference to a vector of something (in our case, its int values), so we 
leave that open what kind of vector the user will come up with. For us, 
it's a vector of t values. The first quick remove at function we used 
accepts index values, which are numbers , so the interface looks like the 
following: 

template ctypename T> 

void quick_remove_at(std::vector<T> &v, std::size_t idx) 

{ 

6. Now comes the meat of the recipe—how do we remove the item quickly 
without moving too many others? First, we simply take the value of the 
last item in the vector and use it to overwrite the item which shall be 
deleted. Second, we cut off the last item of the vector. These are the two 
steps. We surround this code with a little sanity check. If the index value 



is obviously out of the vector range, we do nothing. Otherwise, the code 
would, for example, crash on an empty vector. 

if (idx < v. size () ) { 

v[idx] = std::move(v.back()); 
v.pop_back(); 

} 

} 

7. The other implementation of quick remove at works similar. Instead of 

/V 

accepting a numeric index, it accepts an iterator forA std: :vector<T>. 
Obtaining its type in a generic manner is not complicated because STL 
containers already define such types. 


template ctypename T> 

void quick_remove_at(std::vector<T> &v, 

typename std::vector<T>::iterator 


{ 


it) 


/v 

8. Now, we willA access the value, at which the iterator is pointing. Just as 

/V 

in the other function, we willA overwrite it by the last element in the 
vector. Because we are handling not a numeric index, but an iterator this 
time, we need to check a bit differently if the iterator position is 

/V 

sane. A If it points to the artificial end position, we are not allowed to 
dereference it. 


if (it != std::end(v)) { 


9. Within that if block, we do the same thing as before—we overwrite the 
item to be removed with the value of the item from the last position- 
then we cut off the last element from the vector: 


*it = std::move(v.back()); 
v.pop_back(); 

} 

} 

10. That's it. Compiling and running the program leads to the following 
output: 


$ ./main 

123, 456, 200, 100, 



100, 456, 200, 


How it works... 

The quick remove at function removes items pretty quickly without 
touching too many other items. It does this in a relatively creative way: It 
kind of swaps the actual item, A which shall be removed with the last item in 
the vector. Although the last item has no connection to the actually selected 
item, it is in a special position : Removing the last item is cheap ! The vector's 
size just needs to be shrunk down by one slot, and that's it. No items are 

/V 

moved during that step. Have a look at the following diagramA which helps 

/\ 

imagingA how this happens: 



1.) initial state 


1 

2 

3 

4 

/ 



I 



begin end 



after quick_remove_at ( vec, iterator) 



iterator 


/\ 

Both the A steps in the recipe code look like this: 

v.at(idx) = std::move(v.back()); 
v.pop_back(); 


/v 

This is the A iterator version, which looks nearly identical: 

*it = std::move(v.back()) ; 
v.pop_back(); 


Logically, we swap the selected item and the last one. But the code does not 
swap items, it moves the last one over the first one. Why? If we swapped the 
items, then we would have to store the selected item in a 





/V 

temporary A variable, move the last item to the selected item, and then store 
the temporary value in the last slot again. This seems useless , as we are just 
about to delete the last item anyway. 

/V 

Ok, fine, so the swap is useless, and a one-way overwrite is aA better thing to 
do. Having seen that, we can argue that this step could also be performed 
with a simple *it = v.back ();, right? Yes, this would be completely 
correct, but imagine we stored some very large strings in every slot, or even 
another vector or map—in that situation, that little assignment would lead to a 

, /V 

very expensive copy. The std: :moveA call in between is just an 

/y 

optimization: A In the example case of strings, the string item internally 
points to a large string in the heap. We do not need to copy that. Instead, 
when moving a string, the destination of the move gets to point at the string 
data of the other. The move source item is left intact, but in a useless state, 
which is fine because we are removing it anyway. 



Accessing std::vector instances the fast or 
the safe way 


The std:: vector is probably the most widely used container in the STL, 
because it holds data just like an array, and adds a lot of comfort around that 
representation. However, wrong access to a vector can still be dangerous. If a 
vector contains 100 elements, and by accident our code tries to access an 
element at index 123, this is obviously bad. Such a program could just crash, 
which might be the best case, because that behavior would make it very 
obvious that there is a bug! If it does not crash, we might observe that the 

/V 

program just behaves strangely A from time to time, which could lead to even 
more headaches than a crashing program. The experienced programmer 
might add some checks before any directly indexed vector access. Such 
checks do not increase the readability of the code, and many people do not 
know that std:: vector already has built-in bound checks! 

How to do it... 

/V 

In this section, we willA use the two different ways to access an 

std:: vector, and then see how we can utilize them to write safer programs 

without decreasing readability. 

1. Let's include all the needed header files, and fill an example vector with 
iooo times the value 123 , so we have something we can access: 

#include <iostream> 

#include <vector> 


using namespace std; 

int main() 

{ 

const size_t container_size {1000}; 
vector<int> v (container size, 123) ; 




2. Now, we access the vector out of bounds using the [ ] operator: 

cout << "Out of range element value: " 

<< v[container_size + 10] << '\n'; 

3. Next, we access it out of bounds using the at function: 

cout << "Out of range element value: " 

<< v.at(container_size + 10) << '\n 1 ; 

} 

4. Let's run the program and see what happens. The error message is GCC 

specific. Other compilers would emit different but similar error 

/\ _ 

messages.A The first read succeeds in a strange way. It doesn't lead the 
program to crash, but it's a completely different value than 123 . We can't 
see the output line of the other access because it purposefully crashed 
the whole program. If that out of bounds access was an accident, we 
would catch it much earlier! 


Out of range element value: -726629391 

terminate called after throwing an instance of 'std::out_of 

what(): array::at: _n (which is 1010) >= _Nm (which is 

Aborted (core dumped) 


How it works... 

/v 

The std:: vector provides theA [ ] operator and the at function, and they 
basically do exactly the same job. The at function, however, performs 
additional bounds checks and throws an exception if the vector bounds are 
exceeded. This is super useful in situations like ours, but also makes the 
program a little bit slower. 

Especially when doing numeric computations with indexed members which 
need to be really fast, it is advantageous to stick to [ ] indexed access. In any 
other situation, the at function helps uncovering bugs with usually negligible 
performance loss. 


Note 



It is good practice to use the at function by default. If the 

/V 

resulting code is too slow but has proven to be bug-free, theA [ ] 
operator can be used in performance-sensitive sections instead. 


There’s more... 


Of course, we can handle out of bounds accesses, instead of letting the whole 
app crash. In order to handle it, we catch the exception, in case it was thrown 
by the at function. Catching such an exception is simple. We just surround 
the at call with a try block and define the error handling in a catch block. 


try { 


} 

} 


std::cout << "Out of range element value: " 

<< v.at(container_size + 10) << '\n 1 ; 
catch (const std::out_of_range &e) { 

std::cout << "Ooops, out of range access detected: 
<< e.what() << '\n'; 


n 


Note 


By the way, std:: array also provides an at function. 



Keeping std::vector instances sorted 


Arrays and vectors do not sort their payload objects themselves. But if we 
need that, this does not mean that we always have to switch to data structures, 
which were designed to do that automatically. If an std:: vector is perfect 
for our use case, it is still very simple and practical to add items to it in a 
sorting manner. 

How to do it... 


In this section, we will fill an std: : vector with random words, sort it, and 
then insert more words while keeping the vector's sorted word order intact. 

1. Let's first include all headers we're going to need. 

#include <iostream> 

#include <vector> 

#include <string> 

#include <algorithm> 

#include <iterator> 

#include <cassert> 

2. We also declare that we are using namespace std in order to spare us 
some std: : prefixes: 

using namespace std; 


3. Then we write a little main function, which fills a vector with some 
random strings. 


int main() 

{ 

vector<string> v 


{"some", "random", "words", 
"without", "order", "aaa", 
"yyy"}; 


4. The next thing we do is sorting that vector. Let's do that with some 




assertions and the is sorted function from the STL before, which 
shows that the vector really was not sorted before, but is sorted 
afterward. 

assert(false == is_sorted(begin(v) , end(v))); 
sort(begin(v) , end(v)); 

assert(true == is_sorted(begin(v), end(v))); 

5. Now, we finally add some random words into the sorted vector using a 

/V 

new A insertsorted function, which we still need to implement 

_ /\ 

afterward. These words shall be put at the right spotA so that the vector 
is still sorted afterward: 

insert_sorted(v, "foobar"); 
insert_sorted(v, "zzz"); 

6. So, let's now implement insert sorted a little earlier in the source file. 

void insert_sorted(vector<string> &v, const string Sword) 

{ 

const auto insert_pos (lower_bound(begin(v), end(v), wo 
v.insert(insert_pos, word); 

} 

7. Now, back in the main function where we stopped, we can now continue 
printing the vector and see that the insert procedure works: 

for (const auto &w : v) { 
cout << w << " "; 

} 

cout << '\n'; 

} 

8. Compiling and running the program yields the following nicely sorted 
output: 

aaa foobar order random some without words yyy zzz 

How it works... 

The whole program is constructed around the insert sorted function, which 



/V 

does what this sectionA is about: For any new string, it locates the position in 
the sorted vector, at which it must be inserted, in order to preserve the order 
of the strings in the vector. However, we assume that the vector was sorted 
before. Otherwise, this would not work. 

The locating step is done by the STL function lower bound, which accepts 

/\ _ 

three A arguments. The first two denote beginning and end of the underlying 

_ -'V 

range. The range is our vector of words in this case. The third argumentA is 
the word, which shall be inserted. The function then finds the first item in the 
range, which is greater than or equal to that third parameter and returns an 
iterator pointing to it. 

Having the right position at hand, we gave it to the std:: vector member 

/V _ 

method insert, which accepts just twoA arguments. The first argument is an 
iterator, which points to the position in the vector, at which the second 
parameter shall be inserted. It appears very handy that we can use the same 
iterator, which just dropped out of the lower bound function. The second 
argument is, of course, the item to be inserted. 

There’s more... 

The insert sorted function is pretty generic. If we generalize the types of 
its parameters, it will also work on other container payload types, and even 

/V 

on other containers such asA std: :set, std::deque, std: : list, and so on! 
(Note that set has its own lower bound member function that does the same 
as std:: lower bound, but is more efficient because it is specialized for sets.) 

template ctypename C, typename T> 
void insert_sorted(C &v, const T &item) 

{ 

const auto insert_pos (lower_bound(begin(v), end(v), item)); 
v.insert(insert_pos, item); 

} 

When trying to switch the type of the vector in the recipe from std:: vector 
to something else, note that not all containers support std:: sort. That 
algorithm requires random access containers, which std:: list, for example, 
does not fulfill. 



Inserting items efficiently and conditionally 
into std::map 


Sometimes we want to fill a map with key-value pairs and while filling the 

/V 

map up, we might run into two A different cases: 

1. The key does not exist yet. Create a fresh key-value pair. 

2. The key does exist already. Take the existing item and modify it. 

/V /V /V 

We could just naively use theA insert or emplace methods ofA mapA and 
see if they succeed. If it doesn't, we have case 2 and modify the existing item. 
In both cases, insert and emplace create the item which we try to insert, and 
in case 2 the freshly created item is dropped. We get a useless constructor call 
in both cases. 

Since C++17, there is the try empiace function, which enables us to create 
items only conditionally upon insertion. Let's implement a program that takes 
a list of billionaires and constructs a map that tells us the number of 
billionaires per country. In addition to that, it stores the wealthiest person in 
every country. Our example will not contain expensive to create items, but 
whenever we find ourselves in such a situation in real-life projects, we know 
how to master it with try empiace. 

How to do it... 

/\ /V 

In this section, we willA implementA an application that creates a map from 
a list of billionaires. The map maps from each country to a reference to the 
wealthiest person in that country and a counter that tells how many 

/V 

billionaires that country has.A 

1. As always, we need to include some headers first and we declare that we 
use namespace std by default. 


#include <iostream> 




#include <functional> 
#include <list> 
#include <map> 

using namespace std; 


2. Let's define a structure that represents billionaire items for our list. 

struct billionaire { 
string name; 
double dollars; 
string country; 

}; 


3. In the main function, we first define the list of billionaires. There are 
many billionaires in the world, so let's construct a limited list with just 
some of the richest persons in some countries. This list is already 

ordered. The rankings are actually taken from the Forbes 2017 list The 

/\ /\ 

World's Billionaires A atA https://www.forbes.com/billionaires/list/: 


int 

{ 


main() 

list<billionaire> billionaires { 

{"Bill Gates", 86.0, "USA"}, 

{"Warren Buffet", 75.6, "USA"}, 

{"Jeff Bezos", 72.8, "USA"}, 

{"Amancio Ortega", 71.3, "Spain"}, 
{"Mark Zuckerberg", 56.0, "USA"}, 
{"Carlos Slim", 54.5, "Mexico"}, 

/ / ... 

{"Bernard Arnault", 41.5, "France"}, 

/ / . . . 

{"Liliane Bettencourt", 39.5, "France" 

/ / ... 

{"Wang Jianlin", 31.3, "China"}, 

{"Li Ka-shing", 31.2, "Hong Kong"} 

// ... 

}; 


f 


4. Now, let's define the map. It maps from the country string to a pair. The 
pair contains a (const) copy of the first billionaire of every country from 
our list. That is automatically the richest billionaire per country. The 
other variable in the pair is a counter, which we will increment for every 
following billionaire in that country. 



map<string, pairCconst billionaire, size_t>> m; 


5. Now, let's go through the list and try to emplace a new payload pair for 
every country. The pair contains a reference to the billionaire we are 
currently looking atA and a counter value of 1 . 

for (const auto &b : billionaires) { 

auto [iterator, success] = m.try_emplace(b.country. 


6. If that step was successful, then we don't need to do anything else. The 
pair for which we provided the constructor arguments b, 1 has been 
constructed and inserted into the map. If the insertion was not successful 
because the country key exists already, then the pair was not 
constructed. If our billionaire structure was very large, this would have 
saved us the runtime cost of copying it. However, in the nonsuccessful 
case, we still need to increment the counter for this country. 

if ( !success) { 

iterator-isecond.second += 1; 

} 

} 

7. Ok, that's it. We can now print how many billionaires there are per 
country, and who is the wealthiest one in each country. 

for (const auto & [key, value] : m) { 

const auto &[b, count] = value; 

cout << b.country << " : " << count 
<< " billionaires. Richest is " 

<< b.name << " with " << b.dollars 
« " B $ \ n " ; 

} 

} 

8. Compiling and running the program yields the following output. (Of 
course, the output is limited, because we limited our input map.) 

$ ./efficient_insert_or_modify 

China : 1 billionaires. Richest is Wang Jianlin with 31.3 B 

France : 2 billionaires. Richest is Bernard Arnault with 41 

Hong Kong : 1 billionaires. Richest is Li Ka-shing with 31. 

Mexico : 1 billionaires. Richest is Carlos Slim with 54.5 B 



Spain : 1 billionaires. Richest is Amancio Ortega with 71.3 
USA : 4 billionaires. Richest is Bill Gates with 86 B$ 


How it works... 

The whole recipe revolves around the try empiace function of std: :map, 
which is a new C++17 addition. It has the following signature: 

std::pairtiterator, bool> try_emplace(const key_type& k, Args&&.. 

/v 

Thus, the key being inserted is parameter kA and the associated value is 
constructed from the parameter pack args. If we succeed in inserting the 
item, then the function returns an iterator , which points to the new node in 
the map, paired with a Boolean value being set to true. If the insertion was 
not successful, the Boolean value in the return pair is set to false, and the 
iterator points to the item with which the new item would clash. 

This characteristic is very useful in our case—when we see a billionaire from 
a specific country for the first time, then this country is not a key in the map 

/V 

yet. In that case, we must insert A it, accompanied with a new counter being 
set to l. If we did see a billionaire from a specific country already, we have to 
get a reference to its existing counter, in order to increment it. This is exactly 
what happened in step 6: 

if (!success) { 

iterator->second.second += 1; 

} 


Note 

Note that both the insert and emplace functions of std: :map 
work exactly the same way. A crucial difference is that 
try empiace will not construct the object associated with the 

/v _ 

keyA if the key already exists. This boosts the performance in 
case objects of that type are expensive to create. 



There’s more... 


The whole program still works if we switch the type of the map from 

std:: map to std:: unordered map. This way, we can simply switch from one 

implementation to another, which has different performance characteristics. 

In this recipe, the only observable difference is that the billionaire map is not 
printed in alphabetical order any longer, because hash maps do not order their 
objects the same way as search trees do. 



Knowing the new insertion hint semantics 
of std::map::insert 


Looking up items in an std: :map takes 0(log(n)) time. This is the same for 
inserting new items, because the position where to insert them must be 
looked up. Naive insertion of M new items would thus take 0(M * log(n)) 
time. 

In order to make this more efficient, std: :map insertion functions accept an 
optional insertion hint parameter. The insertion hint is basically an iterator, 

/V 

which points near the future position of the item thatA is to be inserted. If the 
hint is correct, then we get amortizedO(l) insertion time. 

How to do it... 

/V 

In this section, we willA insert multiple items into an std: :map, and use 
insertion hints for that, in order to reduce the number of lookups. 

1. We are mapping strings to numbers, so we need the header files 
included for std: :map and std: : string. 

#include <iostream> 

#include <map> 

#include <string> 

2. The next step is to instantiate a map, which already contains some 
example characters. 

int main() 

{ 

std::map<std::string, size_t> m {{"b", 1}, {"c", 2}, {" 

/v /v 

3. We willA insert multiple items now, and for each item we willA use an 
insertion hint. Since we have no hint in the beginning to start with, we 
will just do the first insertion pointing to the end iterator of the map. 




auto insert it (std::end(m)) ; 


4. We will now insert items from the alphabet backward while always 
using the iterator hint we have, and then reinitialize it to the return value 
of the insert function. The next item will be inserted just before the 
hint. 


for (const auto &s : {"z", "y", "x", "w"}) { 

insert_it = m.insert(insert_it, {s, 1}); 

} 

5. And just for the sake of showing how it is not done, we insert a string 
which will be put at the leftmost position in the map, but give it a 
completely wrong hint, which points to the rightmost position in the 
map—the end. 


m.insert(std::end(m) , {"a", 1}); 

6. Finally, we just print what we have. 

for (const auto & [key, value] : m) { 

std::cout << << key << " << value << ", " 

} 

std::cout << '\n'; 


7. And this is the output we get when we compile and run the program. 
Obviously, the wrong insertion hint did not hurt too much, as the map 
ordering is still correct. 


1, "b": 1, 


2, "d": 3, 


w 


1 , 


X 


1, "y": 1, 


How it works... 


The only difference to normal map insertions in this recipe was the additional 
hint iterator. And we spoke about correct and wrong hints. 

A correct hint will point to an existing element, which is greater than the 

/\ 

element to be insertedA so that the newly inserted key will be just before the 
hint. If this does not apply for the hint the user provided during an insertion, 



the insert function will fall back to a nonoptimized insertion, yielding 
0(log(n)) performance again. 


For the first insertion, we got the end iterator of the map, because we had no 
better hint to start with. After installing a "z" in the tree, we knew that 
installing "y" will insert a new item just in front of the "z", which qualified it 
to be a correct hint. ThisA applies to "x" as well, if put into the tree after 
inserting the "y", and so on. This is why it is possible to use the iterator, 
which was returned by the last insertion for the next insertion. 

Note 

It is important to know, that before C++11, insertion hints were 

/V 

considered correctA when they pointed before the position of the 
newly inserted item. 

There’s more... 

Interestingly, a wrong hint does not even destroy or disturb the order of the 
items in the map, so how does that even work, and what did that mean, that 
the insertion time is amortized 0(1)7 

The std:: map is usually implemented using a binary search tree. When 
inserting a new key into a search tree, it is compared against the keys of the 
other nodes, beginning from the top. If the key is smaller or larger than the 
key of one node, then the search algorithm branches left or right to go down 
to the next deeper node. While doing that, the search algorithm will stop at 
the point where it reached the maximum depth of the current tree, where it 
will put the new node with its key. It is possible that this step destroyed the 
tree's balance, so it will also correct that using a re-balancing algorithm 

/V 

afterwardA as a housekeeping task. 

When we insert items into a tree with key values which are direct neighbors 
of each other (just as the integer 1 is a neighbor of the integer 2 , because no 
other integer fits between them), they can often also be inserted just next to 
each other in the tree, too. It can easily be checked if this is true for a certain 



key and an accompanying hint. And if this situation applies, the search 
algorithm step can be omitted, which spares some crucial runtime. Afterward, 
the re-balancing algorithm might nevertheless have to be run. When such an 
optimization can often be done, but not always , this can still lead to an 
average performance gain. It is possible to show a resulting runtime 
complexity which settles down after multiple insertions, and then it's called 
amortized complexity. 
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If the insertion hin t is wrong, the insertion function will simply waive the hin t 
and start over using the search algorithm. This works correctlyA but is 
obviously slower. 






Efficiently modifying the keys of std::map 
items 


Since the std: :map data structure maps from keys to values in a way that the 
keys are always unique and sorted, it is of crucial value that users cannot 
modify the keys of map nodes that are already inserted. In order to prevent 
the user from modifying the key items of perfectly sorted map nodes, 

/V 

theA constA qualifier is added to the key type. 

/v 

This kind of restriction is perfectly sane because itA makes it harder for the 

/V /V 

user toA use std: :map the wrong way. A But what shall we do if we really 
need to change the keys of some map items? 

_ /V 

Prior to A C++17, we had to remove the items of which we need to change 
the key value from the tree, in order to reinsert them. The downside of this 
approach is that this always needlessly reallocates some memory, which 
sounds bad in terms of performance. 

/\ 

Since C++17, we can remove and reinsert map nodes without A any 

/\ /\ 

reallocation of memory. We willA see how that works in this recipe. A 

How to do it... 

We implement a little application that orders the placement of drivers in a 
fictional race in an std: :map structure. While drivers pass each other during 
the race, we need to changeA their placement keys, which we do the new 
C++17 way. 

1. Let's first include the necessary headers and declare that we use 
namespace std. 


#include <iostream> 
#include <map> 




using namespace std; 


/v 

2. We willA print the race placements before and after manipulating the 
map structure, so let's implement a little helper function for that. 


template <typename M> 
void print(const M &m) 

{ 

cout << "Race placement:\n"; 

for (const auto &[placement, driver] : m) { 

cout << placement << " << driver << 1 \n 1 ; 

} 

} 

3. In the main function, we instantiate and initialize a map that maps from 
integer values that denote the driver's place to strings that contain the 

/V 

driver's name. We also print the map because we will A modify it in the 
next steps. 

int main() 

{ 

map<int, string> race_placement { 

{1, "Mario"}, {2, "Luigi"}, {3, "Bowser"}, 

{4, "Peach"}, {5, "Yoshi"}, {6, "Koopa"}, 

{7, "Toad"}, {8, "Donkey Kong Jr."} 

}; 


print(race_placement); 

4. Let's say that during one race lap, Bowser had a little accident and 
dropped to the last place and Donkey Kong Jr. took the chance to jump 
from the last place to the third place. In that case, we first need to extract 
their map nodes from the map because this is the only way to manipulate 
their keys. The extract function is a new C++17 feature. It removes 
items from a map without any allocation-related side effects. Let's also 
open a new scope for this task. 

{ 

auto a (race_placement.extract(3)); 
auto b (race_placement.extract(8)); 


5. Now we can swap Bowser's and Donkey Kong Jr.'s keys. While the keys 
of map nodes are usually not mutable because they are declared const, 



we can modify the keys of items which we extracted using the extract 
method. 

swap(a.key(), b.key()); 

/v 

6. std:: map's insert method gotA a new overload in C++17 that accepts 
the handles of extracted nodes, in order to insert them without touching 
the allocator. 

race_placement.insert(move(a)); 
race_placement.insert(move (b) ) ; 

} 

7. After leaving the scope, we're done. We print the new race placement 
and let the application terminate. 

print(race_placement); 

} 

8. Compiling and running the program yields the following output. We see 
the race placement in the fresh map instance first, and then we see it 
again after swapping Bowser's and Donkey Kong Jr.’s positions. 

$ ./mapnode_key_modification 
Race placement: 

1: Mario 

2 : Luigi 

3 : Bowser 

4 : Peach 
5: Yoshi 
6: Koopa 
7 : Toad 

8: Donkey Kong Jr. 

Race placement: 

1: Mario 
2: Luigi 

3: Donkey Kong Jr. 

4 : Peach 
5: Yoshi 
6: Koopa 

7 : Toad 

8 : Bowser 





How it works... 


In C++17, std:: map got a new member function extract. It comes in 

/\ 

two A flavors: 

node_type extract(const_iterator position); 
node_type extract (const key_type& x); 


In this recipe, we used the second one, which accepts a key and then finds 
and extracts the map node that matches the key parameter. The first one 

/V 

accepts an iterator, which implies that it is faster A because it doesn't need to 
search for the item. 

If we try to extract an item that doesn't exist with the second method (the one 

/V 

that searches using a key),A it returns an emptynode_type instance. 

/V 

TheA empty () member method returns us a Boolean value that tells 

/V 

whetherA a node type instance is empty or not. Accessing any other method 
on an empty instance leads to undefined behavior. 

After extracting nodes, we were able to modify their keys using the key () 
method, which gives us nonconst access to the key, although keys are usually 
const. 

Note that in order to reinsert the nodes into the map again, we had to move 

_ /\ 

them into the insert function. This makes senseA because extract is all 
about avoiding unnecessary copies and allocations. Note that while we move 
a nodetype instance, this does not result in actual moves of any of the 
container values. 


There’s more... 

/V 

Map nodes thatA have been extracted using the extract method are actually 

/V /\ 

very versatile. We can extract nodes A from aA map instance and insert it into 
any other map or even multimap instance. It does also work between 

/V 

unordered_map and unordered_multimapA instances, as well as 

/v 

withA set/multiset and respective unordered_set/unordered_multiset. 



In order to move items between different map/set structures, the types of key, 
value, and allocator need to be identical. Note that even if that is the case, we 
cannot move nodes from a map to an unordered map, or from a set to an 

unordered set. 



Using std::unordered_map with custom 
types 


If we use std:: unordered_map instead of std: :map, we have a different 
degree of freedom for the choice of the key type which shall be used, 
std::map demands that there is a natural order between all key items. This 
way, items can be sorted. But what if we want, for example, mathematical 
vectors as a key type? There is no meaning in a smaller< relation for such 
types, as a vector (o, i) is not smaller or larger than ( 1 , o). They just point 
in different directions. This is completely fine for std: :unordered_map 
because it will not distinguish items via their smaller/greater ordering 

/V 

relationshipA but via hash values. The only thing we need to do is to 
implement a hash function for our own type, and an equal to== operator 

implementation, which tells whether A two objects are identical. This 

/\ 

sectionA will demonstrate this in an example. 

How to do it... 

/V 

In this section, we willA define a simple coord struct, which has no default 
hash function, so we need to define it ourselves. Then we put it to use by 
mapping coord values to numbers. 

1. We first include what's needed in order to print and use 

std::unordered_map. 

#include <iostream> 

#include <unordered_map> 

2. Then we define our own custom struct, which is not trivially hashable by 
existing hash functions: 


struct coord { 
int x; 
int y; 




}; 

3. We do not only need a hash function in order to use the structure as a 
key for a hash map, it also needs a comparison operator implementation: 

bool operator==(const coord &1, const coord &r) 

{ 

return 1.x == r.x && l.y == r.y; 

} 


4. In order to extend the STL's own hashing capabilities, we willA open the 
std namespace and create our own std: :hash template struct 
specialization. It contains the same using type alias clauses as other 
hash specializations. 


namespace std 
{ 

template <> 
struct hash<coord> 

{ 

using argument_type = coord; 
using result_type = size_t; 

5. The meat of this struct is the operator () definition. We are just 
adding the numeric member values of struct coord, which is a poor 
hashing technique, but for the sake of showing how to implement it, it's 
good enough. A good hash function tries to distribute values as evenly 
over the whole value range as possible, in order to reduce the amount of 
hash collisions. 


}; 


result_type operator()(const argument_type &c) const 
{ 

return static_cast<result_type>(c.x) 

+ static_cast<result_type>(c.y); 

} 


} 

6. We can now instantiate a new std: :unordered_map instance, which 
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accepts struct coord instances as a key, and maps itA to arbitrary 
values. As this recipe is about enabling our own types for 



std:: unordered_map, this is pretty much it already. Let's instantiate a 
hash-based map with our own type, fill it with some items, and print its : 


int main() 

{ 


std::unordered_map<coord, int> m {{{0, 0}, 1}, {{0, 1}, 

{{2, 1}, 3}}; 

for (const auto & [key, value] : m) { 

std::cout << "{(" << key.x << ", " << key.y 

<< "): " << value << "} 

} 

std::cout << '\n' ; 


7. Compiling and running the program yields the following output: 

$ ./custom_type_unordered_map 
{ (2, 1) : 3} { (0, 1) : 2} { (0, 0) : 1} 

How it works... 


Usually, when we instantiate a hash-based map implementation like 

std: :unordered_map, we write: 

std::unordered_map<key_type, value_type> my_unordered_map; 


It is not too obvious that there happens a lot of magic in the 
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backgroundA when the compiler creates our std: :unordered_map 
specialization. So, let's have a look at the complete template-type definition 
of it: 

template< 

class Key, 
class T, 

class Hash = std::hash<Key>, 

class KeyEqual = std::equal_to<Key>, 

class Allocator = std::allocator< std::pair<const Key, T> > 
> class unordered_map; 

/v 

The first twoA template types are those we filled with coord and int, which 



/v 

is the simple and obvious part. The other threeA template types are optional, 
as they are automatically filled with existing standard template classes, which 
themselves take template types. Those are fed with our choice for the first 

/V 

two A parameters as default values. 

Regarding this recipe, the class Hash template parameter is the interesting 

one: when we do not explicitly define anything else, it is going to be 

specialized on std: :hash<key_type>. The STL already contains std: :hash 

/\ 

specializations for a lot of types such asA std: :hash<std: : string>, 
std: :hash<int>, std::hash<unique_ptr>, and many more. These classes 
know how to deal with such specific types in order to calculate optimal hash 
values from them. 

However, the STL does not know how to calculate a hash value from our 

- /V 

struct coord, yet. So what we did was to just defineA another 
specialization, which knows how to deal with it. The compiler can now go 
through the list of all std: :hash specializations it knows, and will find our 
implementation to match it with the type we provided as key type. 

If we did not add a new std: :hash<coord> specialization, and named it 
my hash type instead, we could still use it with the following instantiation 
line: 

std::unordered_map<coord, value_type, my_hash_type> my_unordered_ 


That is obviously more to type, and not as nice to read as when the compiler 
finds the right hashing implementation itself. 



Filtering duplicates from user input and 
printing them in alphabetical order with 
std::set 


std:: set is a strange container: It kind of works like std: :map, but it 
contains only keys as values, no key-value pairs. So it can hardly be used as a 
way to map values of one type to the other. Seemingly, just because there are 
less obvious use cases for it, a lot of developers do not even know about its 
existence. Then they start to implement things themselves, although 
std:: set would be of great help in some of these situations. 

, /v 

This sectionA shows how to put std:: set to use in an example where we 
collect potentially many different items, in order to filter them and output a 
selection of the unique ones. 


How to do it... 


/V 

In this section, we willA read a stream of words from the standard input. All 
unique words are put into an std:: set instance. This way we can then 
enumerate all unique words from the stream. 

/V _ 

1. We willA use several different STL types, for which we need to include 
multiple headers. 

#include <iostream> 

#include <set> 

#include <string> 

#include <iterator> 


/v 

2. In order to spare us some typing, we willA declare that we are using 
namespace std: 


using namespace std; 




3. Now we're already writing the actual program, which begins with the 
main function instantiating an std:: set which stores strings. 

int main() 

{ 

set<string> s; 


4. The next thing to do is to get the user input. We’re just reading from 
standard input, and do that using the handy istream iterator. 

istream_iterator<string> it {cin}; 
istream_iterator<string> end; 


/v 

5. Having a pair of beginA and end iterators, which represent the user 
input, we can just fill the set from it using an std:: inserter. 

copy(it, end, inserter (s, s.end())); 

6. That's already it. In order to see what unique words we got from 
standard input, we just print the content of our set. 


for (const auto word : s) { 
cout << word << ", 

} 

cout << '\n'; 

} 

/v 

7. Let's compile and run our program with the following input.A We get 
the following output for the preceding input, where all duplicates are 
stripped out, and the words which were unique, are sorted 
alphabetically. 

$ echo "a a a b c foo bar foobar foo bar bar" | ./program 
a, b, bar, c, foo, foobar. 

How it works... 


This program consists of two interesting parts. The first part is using 
std:: istream iterator to access the user input, and the second part is to 
combineA this with our std:: set instance using the std:: copy algorithm, 



after we wrapped it into an std:: inserter instance! It might look surprising 
that there is only one line of code which does all the work of tokenizing the 
input, putting it into the alphabetically sorted set, and dropping all duplicates. 

std: :istream_iterator 

This class is really interesting in cases where we want to process masses of 

data of the same type from a stream, which is exactly the case in this recipe: 

/\ 

we parse the whole input word by wordA and put it into the set in the form of 
std: : string instances. 

The std:: istream iterator takes one template parameter. That is the type 
of the input we want to have. We chose std:: string because we assume text 
words, but it could also have been float numbers, for example. It can 
basically be every type for which it is possible to write cin » var; . The 
constructor accepts an istream instance. The standard input is represented by 
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the global input stream object std:: cin, which is anA acceptable istream 
parameter in this case. 

istream_iterator<string> it {cin}; 

The input stream iterator it which we have instantiated, is able to do two 
things: when it is dereferenced (*it), it yields the current input symbol. As 

we have typed the iterator to std:: string via its template parameter, that 
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symbol will be a string containing one word. A When it is incremented 
(++it), it will jump to the next word, which we can access by dereferencing 
again. 

/V 

But wait, we need to be careful after every incrementA before we dereference 
it again. If the standard input ran empty , the iterator must not be dereferenced 
again. Instead, we should terminate the loop in which we dereference the 
iterator to get at every word. The abort condition, which lets us know that the 
iterator became invalid, is a comparison with the end iterator. If it == end 
holds, we are past the end of the input. 

We create the end iterator by creating an std:: istream iterator instance 
with its parameterless standard constructor. It has the purpose of being the 
counterpart of the comparison which shall act as the abort condition in every 



iteration: 


istream_iterator<string> end; 


As soon as std:: cin runs empty, our it iterator instance will notice that and 
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make a comparison with end returningA true. 

std:: inserter 

We used the it and end pair as input iterators in the std:: copy call. The third 
parameter must be an output iterator. For that, we cannot just take s .begin () 
or s . end (). In an empty set, both are the same, so we are not even allowed to 
dereference it, regardless if that is for reading from it or assigning to it. 

This is where std:: inserter comes into play. It is a function which returns 

/v 

an std:: insert iterator that behaves like an iteratorA but does something 
else than what usual iterators do. When we increment it, it does nothing. 

When we dereference it and assign something to it, it will take the container 
it is attached to, and insert that value as a new item into it! 


When instantiating an std: : insert_iterator via std: : inserter, two 

parameters are needed: 

auto insert_it = inserter(s, s.end() ) ; 

The s is our set, and s. end ( ) is an iterator thatA points to where the new 
item shall be inserted. For an empty set which we start with, this makes as 
much sense as s .begin ( ). When used for other data structures as vectors or 
lists, that second parameter is crucial for defining where the insert iterator 
shall insert new items. 

Putting it together 

In the end, all the action happens during the std:: copy call: 

copy(input_iterator_begin, input_iterator_end, insert_iterator); 

This call pulls the next word token out of std:: cin via the input iterator and 



pushes it into our std:: set. Afterward, it increments both iterators, and 

/V 

checks whetherA the input iterator is equal to the input end iterator 
counterpart. If it is not, then there are still words left in the standard input, 
it will repeat. 

Duplicate words are automatically dropped. If the set already contains a 
specific word, adding it again has no effect. This would be different in an 
std:: multi set A as, in contrast, it would accept duplicates. 



Implementing a simple RPN calculator with 
std::stack 


The std:: stack is an adapter class which lets the user push objects onto it 
like on a real stack of objects, and pop objects down from it again. In this 
section, we construct a reverse polish notation (RPN) calculator around that 
data structure, in order to show how to use it. 

The RPN is a notation thatA can be used to express mathematical expressions 

/V 

in a way thatA is really simple to parse. In RPN, 1 + 2 is 1 2 +. Operands 

/\ 

first, then the operation. Another example:A ( 1 + 2 ) * 3 would be 1 2 + 3 
* in RPN and that already shows why it is easier to parse, as we do not need 
parentheses to define subexpressions. 

RPN text Mathematical 

forum expression tree 











How to do it... 


/V 

In this section, we willA read a mathematical expression in RPN from the 

/\ 

standard input, and then feed it into a function thatA evaluates it. In the end, 
we print the numeric result back to the user. 

/V _ 

1. We willA use a lot of helpers from the STL, so there are a few includes: 

#include <iostream> 

#include <stack> 

#include <iterator> 

#include <map> 

#include <sstream> 

#include <cassert> 

#include <vector> 

#include <stdexcept> 

#include <cmath> 

2. And we do also declare that we are using namespace std in order to 
spare us some typing. 


using namespace std; 


3. Then, we immediately start implementing our RPN parser. It will accept 
an iterator pair, which denotes the beginning and end of a mathematical 
expression in string form, which will be consumed token by token. 


template <typename IT> 

double evaluate_rpn(IT it, IT end) 

{ 

4. While we iterate through the tokens, we need to memorize all operands 
on the way until we see an operation. This is where we need a stack. All 
the numbers will be parsed and saved in double precision floating point, 
so it's going to be a stack of double values. 


stack<double> val stack; 


5. In order to comfortably access elements on the stack, we implement a 
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helper. It modifies the stack by pulling the highest item from itA and 
then returns that item. This way we can perform this task in one single 



step later. 

auto pop_stack ([&](){ 

auto r (val_stack.top()); 
val_stack.pop(); 
return r; 

}) ; 

6. Another preparation is to define all the supported mathematical 
operations. We save them in a map, which associates every operation 
token with the actual operation. The operations are represented by 
callable lambdas, which take two operands, add or multiply them, for 
example, and then return the result. 

map<string, double (*)(double, double)> ops { 



[] 

(double 

a. 

double 

b) 

{ 

return 

a + b; 

11, 


[] 

(double 

a. 

double 

b) 

{ 

return 

a - b; 

11, 

J H * H 

[] 

(double 

a. 

double 

b) 

{ 

return 

a * b; 

11, 


[] 

(double 

a. 

double 

b) 

{ 

return 

a / b; 

11, 

{" A ", 

[] 

(double 

a. 

double 

b) 

{ 

return 

pow(a. 

b) ; 

r if o, n 

l ° f 

[] 

(double 

a. 

double 

b) 

{ 

return 

fmod(a. 

b) 


}; 

7. Now we can finally iterate through the input. Assuming that the input 
iterators give us strings, we feed a new std::stringstream per token, 
because it can parse numbers. 

for (; it != end; ++it) { 

stringstream ss {*it}; 

8. Now with every token, we try to get a double value out of it. If that 
succeeds, we have an operand , which we push on the stack. 

if (double val; ss >> val) { 
val_stack.push(val); 

} 

/v 

9. If it does not succeed, it must be something otherA than an operator; in 

that case, it can only be an operand. Knowing that all the operations we 

/\ 

support are binary, we need to pop the last two A operands from the 
stack. 



else { 

const auto r {pop_stack() } ; 

const auto 1 {pop_stack()}; 

10. Now we get the operand from dereferencing the iterator it, which 
already emits strings. By querying the ops map, we get a lambda object 
which accepts the two operands 1 and r as parameters. 

try { 

const auto & op (ops.at(*it)); 

const double result {op(l, r)}; 
val_stack.push(result); 

} 

11. We surrounded the application of the math part with a try clause, so we 
can catch possibly occurring exceptions. The at call of the map will 
throw an out of range exception in case the user provides a 
mathematical operation we don't know of. In that case, we will rethrow a 

/V 

different exception, which says invalid argument A and carries the 
operation string which was unknown to us. 

catch (const out_of_range &) { 

throw invalid_argument(*it); 

} 

12. That's already it. As soon as the loop terminates, we have the final result 
on the stack. So we return just that. (At this point, we could assert if the 
stack size is 1. If it wasn't, then there would be missing operations.) 

} 

} 

return val_stack.top(); 

} 

13. Now we can use our little RPN parser. In order to do this, we wrap the 
standard input into an std: : istream iterator pair, and feed that into 
the RPN parser function. Finally, we print the result: 

int main() 

{ 


try { 



cout << evaluate_rpn(istream_iterator<string>{cin}, 
« '\n'; 


/v 

14. We do again have that line wrapped into a try clauseA because there's 
still the possibility that the user input contains operations we did not 
implement. In that case, we must catch the exception which we throw in 
such cases, and print an error message: 

catch (const invalid_argument &e) { 

cout << "Invalid operator: " << e.what() << 1 \n 1 ; 

} 

} 

15. After compiling the program, we can play around with it. The input "3 l 

/\ 

2 + * 2 /"A represents the expression ( 3 * (l + 2 ) ) / 2 and 
yields the correct result: 

$ echo "3 1 2 + * 2 /" | ./rpn_calculator 

4.5 

How it works... 

—.r-, /V 

The whole recipe revolves around pushing operands onto the stackA until we 
see an operation in the input. In that situation, we pop the last two operands 
from the stack again, apply the operation to them, and push the result onto the 
stack again. In order to understand all of the code in this recipe, it is 
important to understand how we distinguish operands and operations from 
the input, how we handle our stack, and how we select and apply the right 
mathematical operation. 

Stack handling 

We push items onto the stack, simply using the push function of std: : stack: 

val_stack.push(val); 

Popping values from it looks a bit more complicated because we 
implemented a lambda for that, which captures a reference to the vai stack 



object. Let's look at the same code, enhanced with some more comments: 

auto pop_stack ([&] () { 

auto r (val_stack.top()); // Get top value copy 
val_stack.pop(); // Throw away top value 

return r; // Return copy 

}) ; 

This lambda is necessary to get the top value of the stackA and remove it 
from there in one step. The interface of std: : stack is not designed in a way 
which would allow doing that in a single call. However, defining a lambda is 
quick and easy, so we can now get values like this: 

double top_value {pop_stack()}; 

Distinguishing operands from operations from user input 

In the main loop of evaiuate rpn, we take the current string token from the 

/\ /V 

iteratorA and then see whetherA it is an operand or not. If the string can be 
parsed into a double variable, then it is a number, and hence also an operand. 
We consider everything which is not easily parseable as a number (such 
asA "+", for example) to be an operation. 

The naked code skeleton for exactly this task is as follows: 

stringstream ss {*it}; 
if (double val; ss >> val) { 

// It's a number! 

} else { 

// It's something else than a number - an operation! 

} 

The stream operator » tells us A if it is a number. First, we wrapped the 
String into an std: : stringstream. Then we USe the stringstream object's 
capability to stream from an std: : string into a double variable, which 
involves parsing. If the parsing faiIs, we know that it does so, because we 
asked it to parse something into a number, which is no number. 


Selecting and applying the right mathematical operation 



After we realize that the current user input token is not a number, we just 

/V 

assume that it is an operation, such asA + or *. Then we query our map, 

which we called ops, to look that operation up and return us a function, which 

/\ 

accepts twoA operands, and returns the sum, or the product, or whatever is 
appropriate. 

The type of the map itself looks relatively complicated: 

map<string, double (*)(double, double)> ops { ... }; 

It maps from string to double (*) (double, double) . What does the latter 
mean? This type description shall read " pointer to a function which takes two 
doubles, and returns a double ". Imagine that the (*) part is the name of the 

/V 

function, such as A in double sum (double, double) , which is immediately 

/\ _ /\ 

easierA to read. The trick here is that our lambdaA [ ] (double, double) { 

/V 

return /* some double */ } A is convertible to a function pointer that 
actually matches that pointer description. Lambdas that don't capture 
anything are generally convertible to function pointers. 

This way, we can conveniently ask the map for the correct operation: 

const auto & op (ops .at(*it)); 

const double result {op(l, r)}; 

The map implicitly does another job for us: If we say ops . at ("too" ), then 
"too" is a valid key value, but we did not store any operation named like this. 
In such a case, the map will throw an exception, which we catch in the recipe. 
We rethrow a different exception whenever we catch it, in order to provide a 
descriptive meaning of this error case. The user will know better what an 

/V 

invalid argument A exception means, compared to an out of 

/\ 

rangeA exception. Note that the user of the evaiuate rpn function might not 
have read its implementation, hence it might be unknown that we are using a 
map inside at all. 


There’s more... 

/V 

As theA evaiuate rpn function accepts iterators, it is very easy to feed it 
with different inputs than the standard input stream. This makes it very easy 



to test, or to adapt to different sources of user input. 

Feeding it with iterators from a string stream or from a string vector, for 
example, looks like the following code, for which evaiuate rpn does not 
have to be changed at all: 


int main() 

{ 

stringstream s {"3 2 1 + * 2 /"}; 

cout << evaluate_rpn(istream_iterator<string>{s}, {}) << '\n 

vector<string> v {"3", "2", "1", "2", 

cout << evaluate_rpn(begin(v), end(v)) << '\n'; 

} 


Note 


Use iterators wherever it makes sense. This makes your code very 
composable and reusable. 



Implementing a word frequency counter 
with std::map 


The std:: map is very useful when categorizing something in order to collect 
statistics about that data. By attaching modifiable payload objects to every 
key which represents an object category, it is pretty simple to implement a 

_ A 

histogram of word frequencies for example. This is what we willA do in this 
section. 


How to do it... 


/V /V 

In this section,A we willA read all user input from standard input, which 
might, for example, be a text file containing an essay. We tokenize the input 
to words, in order to count which word occurs how often. 

1. As always, we need to include all the headers from the data structures 
we are going to use. 

#include <iostream> 

#include <map> 

#include <vector> 

#include <algorithm> 

#include <iomanip> 

2. To spare us some typing, we declare that we use namespace std. 


using namespace std; 


3. We will use one helper function in order to crop possibly appended 
commas, dots, or colons from words. 

string filter_punctuation(const string &s) 

{ 

const char *forbidden "}; 

const auto idx_start (s.find_first_not_of(forbidden)); 
const auto idx end (s.find last not of(forbidden)); 




return s.substr(idx_start, idx_end - idx_start + 1); 

} 

4. Now we start with the actual program. We will collect a map 
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thatA associates every word we see with a counter of that word's 
frequency. Additionally, we maintain a variable which records the size 

/V 

of theA longest word we've seen so far, so we can indent the word 

/\ 

frequency table nicelyA when we print it at the end of the program. 

int main() 

{ 

map<string, size_t> words; 
int max_word_len {0}; 

5. When we stream from std:: cin into an std:: string variable, the input 
stream will cut out white space on the way. This way we get the input 
word by word. 

string s; 

while (cin >> s) { 

_ /\ 

6. The word which we have now, could contain a comma, dots,A or a 

colon, because it might be at the end of a sentence or similar. We filter 
that out with the helper function we defined before. 

auto filtered (filter_punctuation(s)); 

7. In case this word is the longest word so far, we need to update the 

max_word_len variable. 

max_word_len = max<int>(max_word_len, filtered.leng 


/v 

8. Now we willA increment the counter value of the word in our words 

/V 

map. If it occurs for the first time, A it is implicitly created before we 
increment it. 

++words[filtered]; 

} 


9. After the loop terminated, we know that we saved all unique words from 



the input stream in the words map, paired with a counter denoting every 
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word's frequency. The map uses words as keysA and is sorted by their 
alphabetical order. What we want is to print all words sorted by their 
frequency, so the words with the highest frequency shall come first. In 
order to get that, we will first instantiate a vector where all these word- 
frequency pairs fit inA and move them from the map to the vector. 

vector<pair<string, size_t>> word_counts; 
word_counts.reserve(words.size ()); 

move(begin(words), end(words), back^inserter(word_count 


10. The vector does now still contain all word-frequency pairs in the same 
order as the words map maintained them. Now we sort it again, in order 
to have the most frequent words at the beginning, and the least frequent 
ones at the end. 


sort(begin(word_counts), end(word_counts), 
[](const auto &a, const auto &b) { 

return a.second > b.second; 

}) ; 


11. All data is in order now, so we push it out to the user terminal. Using the 
std:: setw stream manipulator, we format the data in aA nicely indented 
format, so it looks kind of like a table. 


} 


cout << "# " << setw(max_word_len) << "<WORD>" << " #<C 
for (const auto & [word, count] : word_counts) { 

cout << setw(max_word_len +2) << word << " #" 

<< count << '\n 1 ; 

} 


12. After compiling the program, we can pipe any text file into it in order to 
get a frequency table. 

$ cat lorem_ipsum.txt | ./word_frequency_counter 

# <WORD> #<COUNT> 
et #574 
dolor #302 
sed #273 
diam #273 
sit #259 
ipsum #259 



How it works... 


/v 

This recipe concentrates on collecting all words in an std: :mapA and then 
shoves all items out of the map and into an std:: vector, which is then sorted 
differently, in order to print the data. Why? 

Let's look at an example. When we count the word frequency in the string "a 
abcbbbdc c ", we would get the following map content: 

a -> 2 
b -> 4 
c -> 3 
d -> 1 

However, that is not the order which we want to present to the user. The 
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program should print b firstA because it has the highest frequency. Then c, 
then a, then d. Unfortunately, we cannot request the map to give us the " key 
with the highest associated value ", then the "key with the second highest 
associated value ", and so on. 

Here, the vector comes into play. We typed the vector to contain pairs of 
strings and counter values. This way it can hold items exactly in the form as 
they drop out of the map. 

vector<pair<string, size_t>> word_counts; 

Then we fill the vector using the word-frequency pairs using the std: :move 
algorithm. This has the advantage that the part of the strings which is 
maintained on the heap will not be duplicated, but will be moved over from 
the map to the vector. This way we can avoid a lot of copies. 

move(begin(words), end(words), back_inserter(word_counts)); 

Note 


Some STL implementations use short string optimization—if the 



/V 

string is not too long, it will not be allocated on the heapA and 
stored in the string object directly instead. In that case, a move is 
not faster. But moves are also never slower! 

The next interesting step is the sort operation, which uses a lambda as a 
custom comparison operator: 


sort(begin(word_counts), end(word_counts), 

[](const auto &a, const auto &b) { return a.second > b.se 

The sort algorithm will take items pairwise, and compare them, which is what 
sort algorithms do. By providing that lambda function, the comparison does 
not just compare if a is smaller than b (which is the default implementation), 
but also compares if a. second is larger than b. second. Note that all objects 
are pairs of strings and their counter values, and by writing a. second we 
access the word's counter value. This way we move all high-frequency words 
toward the beginning of the vector, and the low-frequency ones to the back. 



Implement a writing style helper tool for 
finding very long sentences in text with 
std::multimap 


Whenever a lot of items shall be stored in a sorted manner, and the key by 
which they are sorted can occur multiple times, std:: multimap is a good 
choice. 

Let's find an example use case: When writing text in German, it is okay to 
use very long sentences. When writing texts in English, it is not. We will 
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implement a tool thatA helps German authors to analyze their English text 
files, focusing on the length of all sentences. In order to help the author in 
improving the text style, it will group the sentences by their length. This way 
the author can pick the longest ones and break them down. 


How to do it... 
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In this section, we willA read all user input from standard input, which we 
will tokenize by whole sentences, and not words. Then we will collect all 
sentences in an std::multimap paired with a variable carrying their length. 
Afterward, we output all sentences, sorted by their length, back to the user. 


1. As usual, we need to include all needed headers, std::multimap comes 
from the same header as std: :map. 


#include 

#include 

#include 

#include 


<iostream> 

<iterator> 

<map> 

<algorithm> 


2. We use a lot of functions from namespace std, so we declare its use 
automatically. 


using namespace std; 




3. When we tokenize strings by extracting what's between dot characters in 
the text, we will get text sentences surrounded by white space such 

asA spaces, new line symbols, and so on. These would increase their 
size in a wrong way, so we filter them out using a helper function, which 
we now define. 

string filter_ws(const string &s) 

{ 

const char *ws {" \r\n\t"}; 
const auto a (s.find_first_not_of(ws)); 
const auto b (s.find_last_not_of(ws)); 
if (a == string::npos) { 

return {}; 

} 

return s.substr(a, b); 

} 

4. The actual sentence length counting function shall take a giant string 
containing all the text, and then return an std: : multimap, which maps 
sorted sentence lengths to the sentences. 

multimap<size_t, string> get_sentence_stats(const string &c 
{ 

5. We begin by declaring the multimap structure, which is intended to be 
the return value, and some iterators. As we will have a loop, we need an 
end iterator. Then we use two iterators in order to point to consecutive 
dots within the text. Everything between is a text sentence. 

multimap<size_t, string> ret; 

const auto end_it (end(content)); 
auto itl (begin(content)); 
auto it2 (find (itl, end__it, 

6. The it 2 will be always one dot further than itl. As long as itl did not 

reach the end of the text, we are fine. The second condition checks 

/\ 

whetherA it2 is really at least some characters further. If that was not 
the case, there would be no characters left to read between them. 


while (itl != end it && distance(itl, it2) > 0) { 



7. We create a string from all characters between the iterators, and filter all 
white space from its beginning and end, in order to count the length of 
the pure sentence. 


string s {filter_ws({it1, it2})}; 

/v 

8. It's possible that the sentence does not contain anything otherA than 
white space. In that case, we simply drop it. Otherwise, we count its 
length by determining how many words there are. This is easy, as there 
are single spaces between all words. Then we save the word count 
together with the sentence in the multimap. 

if (s.length() > 0) { 

const auto words (count(begin(s), end(s), 1 1 ) 

ret.emplace(make_pair(words, move(s))); 

} 


9. For the next loop iteration, we put the leading iterator iti on the next 
sentence's dot character. The following iterator it2 is put one character 
after the old position of the leading iterator. 

itl = next(it2, 1); 
it2 = find(itl, end_it, 

} 

10. After the loop is terminated, the multimap contains all sentences paired 
with their word count and can be returned. 


return ret; 

} 

11. Now we put the function to use. First, we tell std :: cin to not skip white 
space, as we want sentences with spaces in one piece. In order to read 
the whole file, we initialize an std:: string from input stream iterators 
which encapsulate std: :cin. 


int main() 

{ 

cin.unsetf(ios::skipws); 

string content {istream iterator<char>{cin}, {}}; 



12. As we only need the multimap result for printing, we put the 

get sentence stats call directly in the loop and feed it with our string. 
In the loop body, we print the items line by line. 


for (const auto & [word_count, sentence] 

: get_sentence_stats(content)) { 

cout << word_count << " words: " << sentence << ".\ 

} 

} 

13. After compiling the code, we can feed the app with text from any text 
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file.A An example Lorem Ipsum text yields the following output. As the 
output is very long for long text with many sentences, it prints the 
shortest sentences first and the longest last. This way we see the longest 
sentences first as terminals usually scroll to the end of the output 
automatically. 

$ cat lorem_ipsum.txt | ./sentence_length 

10 words: Nam quam nunc, blandit vel, luctus pulvinar, 
hendrerit id, lorem. 

10 words: Sed consequat, leo eget bibendum sodales, 
augue velit cursus nunc,. 

12 words: Cum sociis natoque penatibus et magnis dis 
parturient montes, nascetur ridiculus mus. 

17 words: Maecenas tempus, tellus eget condimentum rhoncus, 
sem quam semper libero, sit amet adipiscing sem neque sed i 


How it works... 


The whole recipe concentrates on breaking down a large string into sentences 
of text, which are assessed for their length, and then ordered in a multimap. 
Because std:: multimap itself is so easy to use, the complex part of the 
program is the loop, which iterates over the sentences: 

const auto end_it (end(content)) ; 

auto itl (begin(content)); // (1) Beginning of string 

auto it2 (find(itl, end^it, // (1) First dot 

while (itl != end_it && std::distance(it1, it2) > 0) { 

string sentence {itl, it2}; 



// Do something with the sentence string... 

itl = std::next(it2, 1); // One character past current ' 

it2 = find(itl, end^it, // Next dot, or end of string 

} 

/v 

Let's look at the code with the following diagramA in mind, which consists of 
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threeA sentences: 


"Lorem ipsum velit cursus. ^ Led sociis. ^ Foobar penatibus augue." 


/ 

V A / 

V A 

itl 

L_iiZ 


JL 

it: 


L 


itl" 




"Loremipsum ^ velitcursus." 

t t t 


3 spaces —> 4 words 



"Led ^sociis." 

t 


1 space 


> 2 words 



"Foobar^ penatibus^, augue." 

t t 

V__ S 

2 spaces —> 3 words 


The itl and it2 are always moved forward through the string together. This 
way they always point to the beginning and end of one sentence. The 

/V 

std:: find algorithm helps us a lot in that regardA because it works like 



"start at theA current positionA and then return an iterator to the next dot 
character. If there is none, return the end iterator 

After we extract a sentence string, we determine how many words it contains, 
so we can insert it into the multimap . We are using the number of words as 
the key for the map nodes, and the string itself as the payload object 
associated with it. There can easily be multiple sentences which have the 
same length. This would render us unable to insert them all into one 
std::map. But since we use std::multimap, this is no problem, because it 
can easily handle multiple keys of the same value. It will keep them all 
ordered in line, which is what we need to enumerate all sentences by their 
length and output them to the user. 

A There’s more... 

After having read the whole file into one large string, we iterate through the 
string and create copies of every sentence again. This is not necessary, as we 
also could have used std:: string view, which will be covered later in this 
book. 

Another way to iteratively get the strings between two consecutive dots is 
std:: regexiterator, which will also be covered in a later chapter of this 
book. 



Implementing a personal to-do list using 
std: :priority_queue 


The std::priority_queue is another container adapter class, such 

/v 

as A std:: stack. It is a wrapper around another data structure (std:: vector 
by default) and provides a queue-like interface for it. This means that items 
can stepwise be pushed into it, and stepwise be popped out of it again. What 

is pushedA into it first, will be popped out of it first. This is usuallyA also 
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abbreviated as a first in, first out (FIFO) queue. A This is the opposite of a 
stack, where the last item pushed onto it is popped out of it first. 

/V 

While we just described the behavior of std:: queue, this sectionA shows 
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how std::priority_queueA works. That adapter is special, as it does not 
only take FIFO characteristics into account A but also mixes it with priorities. 
This means that the FIFO principle is kind of broken down into sub-FIFO 
queues, which are ordered by the priorities their items have. 

How to do it... 


/V 

In this section, we willA set up a cheap to-do list organizing structure. We do 
not parse user input in order to keep this program short and concentrate on 
std:: prior ity_queue. So we're just filling an unordered list of to-do items 

with priorities and descriptions into a priority queue, and then read them out 
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like from a FIFO A queue data structure, but grouped byA the priorities of the 
individual items. 

1. We need to include some headers first, std: :priority_queue is in the 
header file < queue>. 

#include <iostream> 

#include <queue> 

#include <tuple> 

#include <string> 




2. How do we store to-do items in the priority queue? The thing is, we 
cannot add items and additionally attach a priority to them. The priority 
queue will try to use the natural order of all items in the queue. We 

could now implement our own struct todo item, and give it a priority 
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number, and a string to-doA description, and then implement the 
comparison operator < in order to make them orderable. 

/V 

Alternatively,A we can just take std: :pair, which enables us to 
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aggregate twoA things in one type and implements comparison for us 
automatically. 

int main() 

{ 

using item_type = std::pair<int, std::string>; 


3. We now have a new type itemtype, which consists of an integer 
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priorityA and a string description. So, let's instantiate a priority queue, 
which maintains such items. 

std::priority_queue<item_type> q; 

/v 

4. We will now A fill the priority queue with different items which have 
different priorities. The goal is to provide an unstructured list, and then 
the priority queue tells us what to do in which order. If there are comics 
to read, and homework to do, of course, the homework must be done 
first. Unfortunately, std: :priority_queue has no constructor, which 
accepts the initializer lists, which we can use to fill the queue from the 
beginning on. (With a vector or a normal list, it would have worked that 
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way.) So we first define the list A and insert it in the next step. 

std::initializer_list<item_type> il { 

{1, "dishes"}, 

{0, "watch tv"}, 

{2, "do homework"}, 

{0, "read comics"}, 

}; 


5. We can now comfortably iterate through the unordered list of to-do 
items and insert them step by step using the push function. 


for (const auto &p : il) { 
q.push(p); 



} 


/V 

6. All items are implicitly sorted, and therefore we have a queue A which 
gives us out items with the highest priority. 

while (! q. empty () ) { 

std::cout << q.top().first << " << q.top().secon 

q•pop(); 

} 

std::cout << '\n'; 

} 

7. Let's compile and run our program. Indeed, it tells us, to do our 
homework first, and after washing the dishes, we can finally watch TV 
and read comics. 

$ ./main 
2: do homework 
1: dishes 
0: watch tv 
0: read comics 


How it works... 

The std: :priority list is very easy to use. We have only used three 
functions: 

1. The q.push (item) pushes an item into the queue. 

2. The q. top () returns a reference to the item which is coming out of the 
queue first. 

/v 

3. The q. pop () A removes the frontmost item in the queue. 

But how did the item ordering work? We grouped a priority integer and a to- 
do item description string into an std: :pair and got automatic ordering. If 
we have an std: :pair<int, std: : string> instance p, we Can write p. first 
to access the integer part, and p. second to access the string part. We did that 
in the loop which prints out all to-do items. 




But how did the priority queue infer that {2 , "do homework" } is more 
important than { 0, "watch tv"}, without us telling it to compare the 
numeric part? 

The comparison operator < handles different cases. Let's assume we compare 
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left < rightA and left and right are pairs. 

1. The left. first != right. first, then it returns left. first < 
right.first. 

2. The left. first == right. first, then it returns left. second < 
right. second. 


This way, we can order the items as we need. The only important thing is that 
the priority is the first member of the pair, and the description is the second 
member of the pair. Otherwise, std: :priority_queue would order the items 
in a way where it looks like the alphabetic order of the items is more 

- A 

important than their priorities. (In that case, watch TV A would be suggested 

/V 

as the first thing to do, and do homeworkA some time later. That would at 
least be great for those of us who are lazy!) 



Chapter 3. Iterators 


We cover the following recipes in this chapter: 

• Building your own iterable range 

• Making your own iterators compatible with STL iterator categories 

• Using iterator wrappers to fill generic data structures 

• Implementing algorithms in terms of iterators 

• Iterating the other way around using reverse iterator adapters 

• Terminating iterations over ranges with iterator sentinels 

• Automatically checking iterator code with checked iterators 

• Building your own zip iterator adapter 



Introduction 


Iterators are an extremely important concept in C++. The STL aims to be as 
flexible and generic as possible, and iterators are a great help in that regard. 
Unfortunately, they are sometimes a bit tedious to use, which is why many 
novices avoid them and fall back to C-Style C++. A programmer who avoids 

_ /V 

iterators basically waives half the potential of the STL. A This chapter deals 

/\ _ /V 

with iterators and quickly casts some light on how theyA work. ThatA very 
quick introduction is probably not enough, but the recipes are really here to 
give a good feeling for iterator internals. 

/V /V 

MostA container classes, but alsoA old-school C-style arrays, in one or the 
other way, contain a range of data items. A lot of day-to-day tasks that 
process a lot of data items do not care how to get at that data. However, if we 
regard, for example, an array of integers and a linked list of integers and want 
to calculate the sum of all the items of both the structures, we would end up 
with two different algorithms, which could look like the following: 

• One algorithm, which deals with the array by checking its size and 
summing it up as follows: 


int sum { 0} ; 

for (size_t i {0}; i < array_size; ++i) { sum += array[i]; 

• Another algorithm, which deals with the linked list by iterating until it 
reaches its end: 

int sum {0} ; 

while (list_node != nullptr) { 

sum += list_node->value; list_node = list_node->next; 

} 

Both of them are about summing up integers, but how large is the percentage 

/\ 

of characters that we typed, which is directly related to the actual A summing 
up task? And does one of them work with a third kind of data structure, let's 
say std:: map, or do we have to implement another version of it? Without 




iterators, this would lead us into ridiculous directions. 

Only with the help of iterators is it possible to implement this in a generic 
form: 


int sum {0} ; 

for (int i : array_or_vector_or_map_or_list) { sum += i; } 

This pretty and short, so-called, range-basedA for loop has been in 

/V 

existenceA since C++11. It is just a syntax sugar, which expands to 

/\ 

somethingA similar to the following code: 

{ 

auto && _range = array_or_vector_or_map_or_list ; 

auto _begin = std::begin(_range); 

auto _end = std::end(_range); 

for ( ; _begin != _end; ++_begin) { 

int i = *_begin; 

sum += i; 

} 

} 

/v 

This is an old hat for everyone who has worked with iterators alreadyA and 
looks completely magic for everyone who didn't. Imagine our vector of 
integers looks like the following: 


vector 


1 

2 

3 

4 

5 


▲ 


vector. begin() vector.endQ 


The std::begin (vector) A command is the same as vector.begin () and 

returns us an iterator that points to the first item (the 1 ). std:: end (vector) is 



the same as vector. end () and returns an iterator that points at one item past 
the last item (past the 5). 

In every iteration, the loop checks if the begin iterator is non-equal to the end 
iterator. If so, it will dereference the begin iterator and thus access the integer 
value it points to. Then, it increments the iterator, repeats the comparison 
against the end iterator, and so on. In that moment, it helps to read the loop 
code againA while imagining that the iterators are plain C-style pointers. In 
fact, plain C-style pointers are also a valid kind of iterators. A 


Iterator categories 

There are multiple categories of iterators, and they have different limitations. 
They are not too hard to memorize, just remember that the capabilities one 
category requires are inherited from the next powerful category. The whole 
point of iterator categories is that if an algorithm knows what kind of iterator 

it is dealing with, it can be implemented in an optimized way. This way, the 

/\ 

programmer can lean back and express his intent, while theA compiler can 

/V 

choose the optimal implementation forA the given task. 


Let's go through them in the right order: 



Iterator category 

Multi pass support 

Defined operations 





Input Iterator 

multiple passes 
not supported 

*it (read-access) 

+ + it or it+ + 




Forward Iterator 


+ + it or it+ + 



Bidirectional Iterator 

multiple passes 
supported 

-it or it- 


Random Access Iterator 

it+ = n or it-=n 

Contiguous Iterator 


contiguous storage 
(like an array) 


Input iterator 

Input iterators can be dereferenced only for reading the values they point to. 
Once they are incremented, the last value they pointed to has been 

invalidated during the incrementation. This means that it is not possible to 

/v 

iterate over such a range multiple times. The A std:: istreamiterator is an 
example for this category. 

Forward iterator 

Forward iterators are the same as input iterators, but they differ in that regard 
that the ranges they represent can be iterated over multiple times. 

/V /\ /V 

TheA std:: forward iistA iterators are an example ofA that. Such a list can 
only be iterated over forward, not backward, but it can be iterated over as 
often as we like to. 


Bidirectional iterator 



















The bidirectional iterator, as the name suggests, can be incremented and 

/V 

decremented, in order to iterate forward orA backward. The iterators 

/\ 

ofA std:: list, std:: set, and std: :map, for example, support that. 


Random access iterator 

Random access iterators allow jumping over multiple values at once, instead 
of single-stepping. This is the case for iterators of std:: vector and 
std::deque. 


Contiguous iterator 

This category specifies all of the aforementioned requirements, plus the 
requirement that the data that is being iterated through lies in contiguous 
memory, like it does in an array, or std:: vector. 

Output iterator 

Output iterators are detached from the other categories. This is because an 
iterator can be a pure output iterator, which can only be incremented and used 
to write to the data it points to. If they are being read from, the value will be 
undefined. 

Mutable iterator 

If an iterator is an output iterator and one of the other categories at the same 
time, it is a mutable iterator. It can be read from and written to. If we obtain 
an iterator from a non-const container instance, it will usually be of this 
kind.A 



Building your own iterable range 


/V 

We already realized that iterators are, kind of, the standard interfaceA for 
iterations over containers of all kin ds. We just need to implement the prefix 

/V /V 

increment operator,A ++, the dereference operator,A *, and the object 

/V 

comparison operator,A ==, and then we already have a primitive iterator that 
fits into the fancy C++11 range-based for loop. 

In order to get used to this a bit more, this recipe shows how to implement an 
iterator that just emits a range of numbers when iterating through it. It is not 
backed by any container structure or anything similar. The numbers are 
generated ad hoc while iterating. 

How to do it... 

In this recipe, we will implement our own iterator class, and then, we will 
iterate through it: 

1. First, we include the header, which enables us to print to the terminal: 

#include <iostream> 

2. Our iterator class will be called num iterator: 

class num_iterator { 

3. Its only data member is an integer. That integer is used for counting. The 

/V 

constructor is for initializing it. It isA generally a good form to make 
constructors explicit , which create a type from another type to avoid 
accidental implicit conversion. Note that we also provide a default value 

_ /V /V 

for position. This makes the instances of theA num iteratorA class 
default-constructible. Although we will not use the default constructor in 
the whole recipe, this is really important because some STL algorithms 
depend on iterators being default-constructible: 






int i; 
public: 

explicit num_iterator(int position = 0) : i{position} { 

4. When dereferencing our iterator (*it), it will emit an integer: 

int operator*() const { return i; } 

5. Incrementing the iterator (++it) will just increment its internal 

/\ 

counter,A i: 

num_iterator& operator++() { 

++i; 

return *this; 

} 

6. A for loop will compare the iterator against the end iterator. If they are 
unequal , it will continue iterating: 

bool operator!=(const num_iterator Sother) const { 
return i != other.i; 

} 

}; 


7. That was the iterator class. We still need an intermediate object for 
writing for (int i : intermediate (a, b) ) {...}, which then 
contains the begin and end iterator, which is preprogrammed to iterate 
from a to b. We call it num_range: 

class num_range { 

8. It contains two integer members, which denote at which number the 
iteration shall start, and which number is the first number past the last 
number. This means if we want to iterate from o to 9, a is set to o and b 
to 10 : 


int a; 
int b; 

public: 

num_range(int from, int to) 
: a{from}, b{to} 



/V /V 

9. There are only two member functions that A we need to implement: A the 

/\ 

begin and end functions. Both return iterators thatA point to the 
beginning and the end of the numeric range: 

num_iterator begin() const { return num^iterator{a}; } 
num iterator end() const { return num iterator{b}; } 


10. That's it. We can use it. Let's write a main function which just iterates 

/V /\ 

over a range thatA goes from 100 to 109 A and prints all its values: 

int main() 

{ 

for (int i : num_range{100, 110}) { 

std::cout << i << ", 

} 

std::cout << 1 \n 1 ; 

} 

11. Compiling and running the program yields the following terminal 
output: 

100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 


How it works... 


Consider that we write the following code: 

for (auto x : range) { code_block; } 


The compiler will evaluate it to the following: 

{ 

auto _begin = std::begin(range); 

auto _end = std::end(range); 

for ( ; _ begin != _end; ++_begin) { 



auto x = *_ begin; 

code block 



While looking at this code, it becomes obvious that the only requirements 

/\ 

forA the iterators are the following three operators: 


• operator ! =: unequal comparison 

• operator++: prefix increment 

• operator*: dereference 

The requirements of the range are that it has a begin and an end method, 
which return two iterators that denote the beginning and the end of a range. 

Note 

In this book, we're mostly using std::begin (x) instead of 

x. begin () . This is generally a good style because 

/\ 

std :: begin (x) automatically calls x. begin () A if that member 
method is available. If x is an array thatA does not have a 
begin o method, std: :begin (x) will automatically find out how 
to deal with it. The same applies to std: : end (x) . User defined 

/V 

types that do not provideA begin () /end () members do not work 

/v 

with std: : begin/std: :end.A 

What we did in this recipe is just fit a simple number counting algorithm into 
the forward iterator interface. Implementing an iterator and a range always 
involves this minimum amount of boilerplate code, which can be a little bit 

/v 

annoying on the one hand. A look atA the loop thatA uses num range is, on 
the other hand, very rewarding because it looks so perfectly simple ! 

Note 

Scroll back and have a thorough look on which of the methods of 
the iterator and the range class are const. Forgetting to make 
those functions const can make the compiler reject your code in 



/\ 

a lot of situationsA because it is a common thing to iterate over 
const objects. 



Making your own iterators compatible with 
STL iterator categories 


Whatever own container data structure we come up with, in order to 
effectively mix it with all the STL goodness, we need to make them provide 
iterator interfaces. In the last section, we learned how to do that, but we do 

/V 

soon realize that some STL algorithms do not compileA well with our custom 
iterators. Why? 

The problem is that a lot of STL algorithms try to find out more about the 
iterators they are asked by us to deal with. Different iterator categories have 
different capabilities, and hence, there might be different possibilities to 

implement the same algorithm. For example, if we copy plain numbers from 

/\ 

one std:: vector to another, thisA may be implemented with a fast 

/\ /\ 

memcpyA call. If we copy data from or toA std:: list, this is not possible any 
longer and the items have to be copied individually one by one. The 
implementers of the STL algorithms put a lot of thought into this kind of 
automatic optimization. In order to help them, we can equip our iterators with 
some information about them. This section shows how to achieve the same. 

How to do it... 

/V /V 

In this section, we will implementA a primitive iterator thatA counts 
numbersA and use it together with an STL algorithm, which initially does not 
compile with it. Then we do what's necessary to make it STL-compatible. 

1. First, we need to include some headers, as always: 

#include <iostream> 

#include <algorithm> 


2. Then we implement a primitive number counting iterator, as in the 




previous section. When iterating over it, it will emit plain increasing 
integers. The num range acts as a handy begin and end iterator donor: 

class num_iterator 

{ 

int i; 
public: 

explicit num_iterator (int position = 0) : i{position} { 

int operator*() const { return i; } 

num_iterator& operator++() { 

_ ++i; 

return *this; 

} 

bool operator!=(const num_iterator &other) const { 
return i != other.i; 

} 

bool operator==(const num_iterator Sother) const { 
return ! (*this != other); 

} 

}; 


class num_range { 
int a; 
int b; 

public: 

num_range(int from, int to) 

: a{from}, b{to} 

{ } 

num^iterator begin() const { return num_iterator{a}; } 
num^iterator end() const { return num_iterator{b}; } 

}; 


3. In order to keep the std: : namespace prefix out and keep the code 
readable, we declare that we use namespace std: 


using namespace std; 



/V 

4. Let's now just instantiate a range thatA goes from 100 to 109 . Note that 

/V 

the value 110 A is the position of the end iterator. This means that no is 
the first number that is outside the range (which is why it goes from 100 
to 109 ): 

int main() 

{ 

num_range r {100, 110}; 

5. And now, we use it withA std: :minmax_eiement. This algorithm 
returns usstd::pair with two members: an iterator pointing to the 
lowest value and another iterator pointing to the highest value in the 
range. These are, of course,A 100 and 109 A because that's how we 
constructed the range: 

auto [min_it, max_it] (minmax_element(begin(r), end(r)) 
cout << *min_it << " - " << *max_it << '\n'; 

} 


6. Compiling the code leads to the following error message. It's some error 

/V 

related toA std:: iterator traits. More on that later. It might happen 
that there are other errors on other compilers and/or STL library 
implementations or no errors at all. This error message occurs with 
clang version 5.0.0 (trunk 299766): 


no type named 'value_type' in 'std::_1::itepatoP_traits<num_itepator>' 

_lessctypename iterator_traits<_ForwardIterator>::value_type>()); 

_a_ 

main.cpp:56:24: in instantiation of function template specialization 'std:: 

_1::minmax_element<num_iterator>' requested here 

auto min_max (std::minmax_element(std::begin(r), std::end(r))); 

A 

1 error generated. 


7. In order to fix this, we need to activate iterator trait functionality for our 
iterator class. Just after the definition of num iterator, we write the 
following template structure specialization of the 

std::iterator_traits type. It tells the STL that our num_iterator is 

of the category forward iterator, and it iterates over int values: 


namespace std { 



template <> 

struct iterator_traits<num_iterator> { 

using iterator_category = std::forward_iterator_tag; 
using value_type = int; 

}; 


} 


8. Let's compile it again; we can see that it works! The output of the 
min/max function is the following, which is just what we expect: 

100 - 109 

How it works... 

Some STL algorithms need to know the characteristics of the iterator type 
they are used with. Some others need to know the type of items the iterators 
iterate over. This has different implementation reasons. 

However, all STL algorithms will access this type information via 
std: : iterator_traits<my_iterator>, assuming that the iterator type is 
my iterator. This traits class contains up to five different type member 
definitions: 


• difference type: What type results from writing itl - it2? 

/\ 

• vaiue type: Of what type is the item which we access with *itA (note 
that this is void for pure output iterators)? 

• pointer: Of what type must a pointer be in order to point to an item? 

• reference: Of what type must a reference be in order to reference an 
item? 

• iterator category: Which category does the iterator belong to? 

The pointer, reference, and difference type type definitions do not make 
sense for our num iterator, as it doesn't iterate over real memory values (we 

/v 

just return int values but they are not persistently availableA like in an 

_ A 

array). Therefore it's better to not define themA because if an algorithm 
depends on those items being referenceable in memory, it might be 



/\ 

buggy A when combined with our iterator. 

There’s more... 


/V 

UntilA C++17, it was encouraged to let iterator types just inherit from 
std:: iterator<... >, which automatically populates our class with all the 
type definitions. This still works, but it is discouraged since C++17. 



Using iterator adaptersA to fill generic data 
structures 


1 


In a lot of situations, we want to fill any container with masses of data, but 
the data source and the container have no common interface. In such a 

/V 

situation, we would need to write our own hand-crafted algorithms thatA just 
deal with the question of how to shove data from the source to the sink. 
Usually, this distracts us from our actual work of solving a specific problem. 

Tasks where we simply transport data between conceptually different data 
structures can be implemented with a one-liner code, thanks to another 
abstraction provided by the STL: iterator adapters. This section 
demonstrates the use of some of them in order to give a feeling how useful 
they are. 

How to do it... 


In this section, we use some iterator wrappers just for the sake of showing 
that they exist and how they can help us in everyday programming tasks. 

1. We need to include some headers first: 

#include <iostream> 

#include <string> 

#include <iterator> 

#include <sstream> 

#include <deque> 

2. Declaring that we use namespace std spares us some typing later: 


using namespace std; 


/v 

3. We start withA std:: istream iterator. We specialize it on int. This 

/\ 

way, it will try to parse the A standard input to integers. For example, if 

/\ 

we iterate over it, it will look as if it wasA std:: vector<int>. The end 



/V 

iterator is instantiated of the same typeA but without any constructor 
arguments: 


int main() 

{ 

istream_iterator<int> it_cin {cin}; 
istream iterator<int> end cin; 


/v /v 

4. Next, we instantiateA std:: deque<int>A and just copy over all the 
integers from the standard input into the deque. The deque itself is not 

/V 

an iterator, SO we wrap it intoA std: :back_insert_iterator using the 
std: : back_inserter helper function. This special iterator wrapper will 
execute v.pushjoack (item) with each of the items we get from the 
standard input. This way the deque is grown automatically! 

deque<int> v; 

copy(it_cin, end_cin, back_inserter(v)); 

/\ 

5. In the next exercise, we useA std:: istrinqstream to copy items into 
the middle of the deque. So, let's first define some example numbers in 
the form of a string and instantiate the stream object from it: 

istringstream sstr {"123 456 789"}; 

_ /v 

6. Then, we need a hint of where to insert into the deque. It will A be the 
middle, so we use the begin pointer of the deque and feed it to the 

std: : next function. The second argument of this function says that it 

/\ 

will return an iterator advanced by v. size () / 2 steps, that is,A half 

/V 

the deque. (We cast v. size () to int A because the second parameter 

/V /V 

ofA std: : next A is dif ference_type of the iterator used as the first 

/V 

parameter. In this case, this A is a signed integer type. Depending on the 
compiler flags, the compiler might warn at this point if we didn't cast 
explicitly.) 

auto deque_middle (next(begin(v), 

static cast<int>(v.size()) / 2 



7. Now, we can copy parsed integers step by step from the input string 

/\ 

streamA into the deque. Again, the end iterator of a stream iterator 

wrapper is just an empty std: : istream_iterator<int> without 

/\ 

constructor arguments (thatA is, the empty {} braces in the code line). 

A The deque is wrapped into an inserter wrapper, which is an 

std:: insert iterator, which is pointed to the deque's middle using 

/\ 

the deque middieA iterator: 

copy(istream_iterator<int>{sstr}, {}, inserter(v, deque 

/v 

8. Now, let's useA std:: front insert iterator to insert some items at 
the front of the deque: 

initializer_list<int> il2 {-1, -2, -3}; 

copy(begin(i!2), end(i!2), front_inserter(v)); 


9. In the last step, we print the whole content of the deque out to the user 

_ /V 

shell. The std::ostream iterator works likeA an output iterator 

which, in our case, just forwards all the integers it gets copied from to 

/\ 

std:: cout A and then appends ", " after each item: 

copy(begin(v), end(v), ostream_iterator<int>{cout, ", " 

cout << 1 \n 1 ; 


10. Compiling and running the program yields the following output. Can 
you identify which number was inserted by which code line? 

$ echo "12345" | ./main 

-3, -2, -1, 1, 2, 123, 456, 789, 3, 4, 5, 

How it works... 


/V 

We used a lot of different iterator adaptersA in this section. They all have one 
thing in common, which is they wrap an object into an iterator that is not an 
iterator itself. 



std: :back_insert_iterator 

/\ 

The back insert iterator can be wrapped aroundA std: : vector, 

std: : deque, std: : list, and SO On. It will Call the container's push back 

/\ 

method, which inserts the new itemA past the existing items. If the container 
instance is not large enough, it will be grown automatically. 

std: :front_insert_iterator 

The front insert_iterator does exactly the same thing as 
back insert iterator, but it calls the container's push front method, 
which inserts the new item before all the existing items. Note that for a 
container like std:: vector, this means that all the existing items need to be 
moved one slot further in order to leave space for the new item at the front. 

std: :insert_iterator 

This iterator adapterA is similar to the other inserters, but is able to insert 

new items between existing ones. The std:: inserter helper function which 

/\ _ 

constructs such a wrapper takes two A arguments. The first argument is the 

/V 

container and the second argument is an iterator thatA points to the position 
where new items shall be inserted. 

std: :istream_iterator 

The istream iterator is another very handy adapter. It can be used with 
any std:: istream object (which can be the standard input or files for 

/V 

example)A and will try to parse the input from that stream object according 
to the template parameter it was instantiated with. In this section, we used 

std: : istream_iterator<int> (std: : cin), which pulls integers Out of the 

standard input for us. 

/V 

The special thing about streamsA is that we often cannot know in advance 
how long the stream is. That leaves the question, where will the end iterator 

point to A if we do not know where the stream's end is? The way this 

/\ 

worksA is that the iterator knows when it reaches the end of the stream. When 



it is compared to the end iterator, it will effectively not really compare itself 
with the end iterator but return if the stream has any tokens left. That's why 
the end iterator constructor does not take any arguments. 

std::ostream iterator 


The ostream iterator is the same thing as the istream iterator, but it 

works the other way around: It doesn't take tokens from an input stream—it 
pushes tokens into an output stream. Another difference to 
istream iterator is that its constructor takes a second argument, which is a 
string that shall be pushed into the output stream after each item. That is 

/V /V /\ 

usefulA because this way we can print a separatingA ", "A or a new line 
after each item. 



Implementing algorithms in terms of 
iterators 


Iterators usually iterate by moving their position from one item of a container 
to another. But they do not necessarily need to iterate over data structures at 
all. Iterators can also be used to implement algorithms, in which case, they 
would calculate the next value when they are incremented (++it) and return 

/V 

that value when they are dereferenced (*it).A 

In this section, we demonstrate this by implementing the Fibonacci function 
in form of an iterator. The Fibonacci function is recursively defined like this: 
f (n) = f (n - l) + F(n - 2). It starts with the beginning values of f (o) = 

A _ 

oA and f ( 1 ) = 1. This leads to the following number sequence: 



F(0) 

= 0 





F(l) 

= l 





F (2) 

= F (1) 

+ 

F (0) 

= 1 


F (3) 

= F (2 ) 

+ 

F (1) 

= 2 


F (4) 

= F (3) 

+ 

F (2) 

= 3 


F (5) 

= F (4 ) 

+ 

F (3) 

= 5 


F ( 6) 

= F (5) 

+ 

F (4) 

= 8 


/V 

A... 

and so on 




/\ 

If we implement this in the form of a callable function thatA returns the 
Fibonacci value for any number, n, we will end up with a recursive self¬ 
calling function, or a loop implementation. This is fine, but what if we write 
some program where have to consume Fibonacci numbers in some pattern, 

/y 

one after the A other? We would have two possibilities—either we recalculate 
all the recursive calls for every new Fibonacci number, which is a waste of 
computing time, or we save the last two Fibonacci numbers as temporary 

/V 

variablesA and use them to calculate the next. In the latter case, we 
reimplemented the Fibonacci algorithm loop implementation. It seems that 
we would end up mixing Fibonacci code with our actual code, which solves a 
different problem: 




size_ta{0} ; 
size_tb{1}; 

for(size_ti{0};i< N;++i){ 
constsize_told_b{b}; 
b+=a; 
a=old_b; 

// do something with b, which is the current fibonacci number 

} 

Iterators are an interesting way out of this. How about wrapping the steps 

/V 

thatA we do in the loop-based iterative Fibonacci implementation in the 
prefix increment ++ operator implementation of a Fibonacci value iterator ? 
This is pretty easy, as this section demonstrates. 

How to do it... 

In this section, we concentrate on implementing an iterator that generates 

/\ 

numbers from the A Fibonacci number sequence while iterating over it. 

1. In order to be able to print the Fibonacci numbers to the terminal, we 
need to include a header first: 

#include <iostream> 

/v 

2. We call the Fibonacci iterator,A f ibit. It will carry a member i, which 
saves the index position in the Fibonacci sequence, and a and b will be 

/V 

the variables thatA hold the last two Fibonacci values. If instantiated 
with the default constructor, a Fibonacci iterator will be initialized to the 

/V 

value A f (o): 

class fibit 
{ 

size_t i {0} ; 
size_t a {0} ; 
size_t b {1}; 

/v 

3. Next, we define the standard constructorA and another constructor, 
which allows us to initialize the iterator at any Fibonacci number step: 



default; 


public: 

fibit() = 

explicit fibit(size_t i_) 
: 1 { 1 _} 

{ } 


4. When dereferencing our iterator (*it), it will just emit the Fibonacci 
number of the current step: 

size_t operator* () const { return b; } 


5. When incrementing the iterator (++it), it will move its state to the next 
Fibonacci number. This function contains the same code as the loop- 
based Fibonacci implementation: 

fibit& operator++() { 

const size_t old_b {b}; 
b += a; 
a = old_b; 

+ + 1; 

return *this; 

} 

6. When used in a loop, the incremented iterator is compared against an 

/\ 

end iterator, for which we need to define the A ! =A operator. We are 
only comparing the step at which the Fibonacci iterators currently 
reside, which makes it easier to define the end iterator for step ioooooo, 
for example, as we do not need to expensively calculate such a high 
Fibonacci number in advance: 

bool operator!=(const fibit &o) const { return i != o.i 

} ; 


7. In order to be able to use the Fibonacci iterator in the range-based for 

loop, we have to implement a range class beforehand. We call it 

/\ 

f ib range, and its constructor will accept one parameter thatA tells how 
far in the Fibonacci range we want to iterate: 


class fib range 



{ 


size_t end_n; 
public: 

fib_range(size_t end_n_) 

: end_n{end_n_} 

{ } 

8. Its begin and end functions return iterators which point to the 

/\ 

positions,A f (0) and f (end_n): 

fibit begin() const { return fibit{}; } 
fibit end() const { return fibit{end n}; } 


9. Okay, now let's forget about all the iterator-related boilerplate code. We 
do not need to touch it again as we have a helper class now which nicely 
hides all the implementation details from us! Let's print the first 10 
Fibonacci numbers: 

int main() 

{ 

for (size_t i : fib_range(10)) { 

std::cout << i << ", 

} 

std::cout << 1 \n 1 ; 

} 

10. Compiling and running the program yields the following shell output: 

1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 

There’s more... 

In order to use this iterator with the STL, it must support the 

std:: iterator traits class. To see how to do that, have a look at the other 

/v 

recipe, which deals with exactly that matter: A Making your own iterators 
compatible with STL iterator categories. 



Note 


Try to think in terms of iterators. This leads to very elegant code 
in many situations. Don't worry about performance: compilers 
find it trivial to optimize away the iterator-related 

/V 

boilerplate A code! 

In order to keep the example simple, we did not do anything about this, but if 

/\ 

we do A publish the Fibonacci iterator as a library, it would become apparent 

/V 

that it has a usability flaw—aA fib it instance that was created with a 
constructor parameter will only be used as an end iterator because it does not 
contain valid Fibonacci values. Our tiny library does not enforce such usage. 
There are differentA possibilities to fix it: 

• Make the fibit (size t i ) constructor private and declare 

/V /V /V /V 

theA fib rangeA class as a friend of theA fibitA class. This way, 
users can only use it the right way. 

• Use iterator sentinels in order to prevent users to dereference the end 
iterator. Have a look at the recipe in which we introduce those: 
Terminating iterations over ranges with iteratorA sentinels 



Iterating the other way around using 
reverse iterator adapters 


Sometimes, it is valuable to iterate over a range the other way around, not 
forward but backward. The range-based for loop, as well as all STL 

algorithms usually iterate over the given ranges by incrementing iterators, 

/\ 

although iterating backwardA requires decrementing them. Of course, it is 
possible to wrap iterators into a layer thatA transforms an increment call 

/V /\ _ 

effectively into aA decrement call. A This sounds like a lot of boilerplate code 
for every type on which we would like to support that. 

_ /V 

The STL provides a helpfulA reverse-iterator adapter, which helps us set up 
such iterators. 


How to do it... 


In this section, we will use reverse iterators in different ways, just to show 
how they are used: 

1. We need to include some headers first, as always: 

#include <iostream> 

#include <list> 

#include <iterator> 

2. Next, we declare that we use namespace std in order to spare us some 
typing: 


using namespace std; 


3. For the sake of having something to iterate over, let's instantiate a list of 
integers: 


int main() 
{ 




list<int> 1 {1, 2, 3, 4, 5}; 


4. Now let's print these integers in the reverse form. In order to do that, we 
iterate over the list by using the rbegin and rend functions of 
std:: list and shove those values out via the standard output using the 
handy ostream iterator adapter: 

copy(1.rbegin(), 1.rend(), ostream_iterator<int>{cout, 
cout << 1 \n 1 ; 


/v 

5. If a container does not provide handy rbegin and rend functionsA but at 
least provides bidirectional iterators, the std: :make_reverse_iterator 
function helps out. It accepts normal iterators and converts them to 
reverse iterators: 

copy(make_reverse_iterator(end(1)) , 

make_reverse_iterator(begin(1) ) , 
ostreany iterator<int> { cout, ", "}); 
cout << 1 \n 1 ; 

} 

6. Compiling and running our program yields the following output: 

5, 4, 3, 2, 1, 

5, 4, 3, 2, 1, 

How it works... 


In order to be able to transform a normal iterator into a reverse iterator, it 

_ /V 

must at least have support for bidirectional iteration. ThisA requirement is 
fulfilled by any iterator of theA bidirectional category or higher. 

A reverse iterator kind of contains a normal iterator and mimics its interface 
completely, but it rewires the increment operation to a decrement operation. 

The next detail is aboutA the begin and end iterator positions. Let's have a 
look at the following diagram, which shows a standard numeric sequence 
kept in an iterable range. If the sequence goes from l to 5, then the begin 



iterator has to point to the element 1 , and the end iterator must point one 

/V 

element pastA 5 : 


i 

i 

i 
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"f 

rend rbegin 

/V 

When defining reverse iterators, the rbegin iterator must point toA 5, and the 
rend iterator must point to the element before A 1 . Turn the book upside 
down, and see that it completely makes sense. 

If we want our own custom container classes to support reverse iteration, we 
do not need to implement all these details ourselves; we can just wrap the 
normal iterators into reverse iterators by using the 

std: :make_reverse iterator helper function, and it does all the operator 
rewiring and offset corrections for us. 






Terminating iterations over ranges with 

/V 

iteratorA sentinels 


Both STL algorithms andA the range-based for loop assume that the begin 
and end positions of the iteration are known in advance. In some situations, 

/V 

however, it is hardly possible to know the end positionA before reaching it 
by iteration. 

A very simple example for this is iterating over plain C-Style strings, the 

/\ /V 

length of which isA not known beforeA runtime. The code which iterates 
over such strings usually looks like this: 

for (const char *c_ponter = some_c_string; *c_pointer != '\0'; ++ 
const char c = *c_pointer; 

// do something with c 

} 

The only way to put this into a range-based for loop seems to be wrapping it 
into an std: : string, which has begin o and end o functions: 

for (char c : std::string(some_c_string)) { /* do something with 

However, the constructor ofstd::string will iterate over the whole string 
before our for loop can iterate over it. Since C++17, we also have 
std:: string view, but its constructor will also iterate through the string 
once. This is not worth the real hassle for shortA strings, but this is also only 

/V 

an example for aA problem class, which can be worth the hassle in other 
situations. TheA std:: istream iterator also has to deal with this when it 
captures input from std:: cin, as its end iterator cannot realistically point 

/V 

toA the end of the user input while the user is still typing keys. 

C++17 comes with the great news that it does not constrain begin and end 

/\ 

iterators to be of the same type. A This section demonstrates how to put this 
little rule change to great use. 




How to do it... 


In this section, we will build an iterator together with a range class, which 
enables us to iterate over a string with unknown length, without finding the 
end position in advance. 

1. First, as always, we need to include headers: 

#include <iostream> 

2. The iterator sentinel is a very central element of this section. 
Surprisingly, its class definition can stay completely empty: 

class cstring_iterator_sentinel {}; 

3. Now we implement the iterator. It will contain a string pointer, which 

/V 

the container A we iterate over: 

class cstring_iterator { 

const char *s {nullptr}; 

4. The constructor just initializes the internal string pointer to whatever 
string the user provides. Let's make the constructor explicit in order to 
prevent accidental implicit conversions from strings to string iterators: 

public: 

explicit cstring_iterator(const char *str) 

: s{str} 

{ } 

5. When dereferencing the iterator at some point, it will just return the 
character value at this position: 

char operator* () const { return *s; } 

6 . Incrementing the iterator just increments the position in the string: 

cstring_iterator& operator++() { 

++s; 

return *this; 

} 



7. This is the interesting part. We implement theA ! = operator for 

comparison, as it is used by STL algorithms and the range-based for 

/\ 

loop. However,A this time, we do not implement it for the comparison 

of iterators with other iterators, but for comparing iterators with 

/\ 

sentinels. When weA compare an iterator with another iterator we can 
only check if their internal string pointers both point to the same 

/V 

address, which is somewhat limiting.A By comparing against an empty 

/\ 

sentinel object, we can performA a completely different semanticawe 
check if the character our iterator points to is a terminating ' \o' 

/V 

characterA because this represents the end of the string! 

bool operator!=(const cstring_iterator_sentinel) const 
return s != nullptr && *s != '\0'; 

} 

}; 


8 . In order to use this in a range-based for loop, we need a range class 
around it, which emits the begin and end iterators: 

class cstring_range { 

const char *s {nullptr}; 

9. The only thing the user needs to provide during instantiation is the string 
that will be iterated over: 


public: 

cstring_range(const char *str) 

: s{str} 

{ } 

/\ 

10. We return a normalA cstring iterator from the begin () function, 
which points to the beginning of the string. From the end () function, we 
just return the sentinel type. Note that without the sentinel type, we 
would also return an iterator, but from where should we know the end of 
the string if we didn't search for it in advance? 

cstring_iterator begin() const { 
return cstring^iterator{s}; 

} 

cstring_iterator_sentinel end() const { 
return {}; 



} 


}; 


11. That's it. We can immediately use it. Strings that come from the user are 

/\ 

one example ofA an input we cannot know the length of in advance. In 
order to force the user to give some input, we will abort the program if 
the user did not provide at least one parameter when launching the 
program in the shell: 

int main(int argc, char *argv[]) 

{ 

if (argc < 2) { 

std::cout << "Please provide one parameter.\n"; 
return 1; 

} 

12. If the program is still being executed up to this point, then we know that 
argv [ l ] contains some user string: 

for (char c : cstring_range(argv[1])) { 

std::cout << c; 

} 

std::cout << 1 \n 1 ; 

} 

13. Compiling and running the program yields the following output: 

$ ./main "abcdef" 
abcdef 


That the loop prints what we just entered is not a surprise, as this is just quite 
a micro-example for the implementation of a sentinel-based iterator range. 

This iteration termination method will help you in implementing your own 

/\ 

iterators wherever you run into aA situation where the comparison with an 

/V 

endpositionA approach is not helpful. 



Automatically checking iterator code with 
checked iterators 


No matter how useful iterators are, and what generic interface they represent, 
iterators can easily be misused , just as pointers. When dealing with pointers, 
code must be written in a way that it never dereferences them when they 
point to invalid memory locations. Same applies to iterators, but there are a 

/V 

lot of rules A that state when an iterator is valid and when it is invalidated. 
Those can easily be learned by studying the STL documentation a bit, butA it 

/V 

will stillA always be possible to write buggy code. 

/\ 

In the best case, such buggy code blows up in front of the developer A while 
it is being tested , and not on the client's machine. However, in many cases, 
the code just silently seems to work, although it dereferences dangling 
pointers, iterators, and so on. In such cases, we want to be eagerly alarmed if 

/V 

we produce codeA showing undefined behavior. 

Fortunately, there's help! The GNU STL implementation has a debug mode , 
and the GNU C++ compiler as well as the LLVM clang C++ compiler both 

/V 

support additional libraries A that can be used to produce extra-sensitive and 
verbose binaries for us, which immediately blow up on a large variety of 
bugs. This is easy to use and super useful, as we will demonstrate in this 
section. The Microsoft Visual C++ standard library also provides a possibility 
to activate additional checks. 


How to do it... 

In this section, we'll write a program thatA deliberately accesses an 
invalidated iterator: 

1. First, we include headers. 


#include <iostream> 
#include <vector> 




2. Now, let's instantiate a vector of integers and get an iterator to the first 
item, the value 1 . We apply shrink to f it () on the vector in order to 
ensure that its capacity is really3, as its implementation might allocate 

/V 

more memory than necessary as a little reserve to makeA future item 
insertionsA faster: 

int main() 

{ 

std::vector<int> v {1, 2, 3}; 
v.shrink_to_fit(); 

const auto it (std::begin(v)); 

3. Then we print the dereferenced iterator, which is completely fine: 

std::cout << *it << 1 \n 1 ; 

4. Next, let's append a new number to the vector. As the vector is not large 

/V 

enough to take another number, it will automaticallyA increase its size. 

/V 

It does this by allocating a new and larger chunkA of memory, 

/V 

movingA all the existing items to that new chunk and then deleting the 
old memory chunk: 

v.push_back(12 3) ; 

/\ 

5. Now, let's printA l from the vector through this iterator again. This is 
bad. Why? Well, when the vector moved all its values to the new chunk 

/V 

of memoryA and threw away the old chunk, it did not tell the iterator 
about this change. This means that the iterator is still pointing to the old 
location, and we cannot know what really happened to it since then: 

std::cout << *it << '\n'; // bad bad bad! 

} 

6. Compiling and running this program leads to a flawless execution. The 
app doesn't crash, but what it prints when dereferencing the invalidated 
pointer is pretty much random. Leaving it like this is pretty dangerous, 
but at this point, no one tells us about that bug if we don't see it 
ourselves: 



$ g++ -std=c++17 nain.cpp -o main 
$ ./nain 
1 


7. Debug flags come to the rescue! The GNU STL implementation 

supports a preprocessor macro called _glibcxx_debug, which activates a 
lot of sanity checking code in the STL. This makes the program slower, 
but it finds bugs. We can activate it by adding a -dglibcxxdebug flag 
to our compiler command line, or define it in the head of the code file 

/V 

before the include lines. As youA can see, it kills the app in the 

mactivate different sanitizers. Let's compile the code with clan useful 

/\ /\ 

(the activationA flag for checked iterators withA the Microsoft Visual 

/V 

C++ compiler isA /d iterator debug level=i):_ 


$ g++ -std=c++17 Plain. cpp -D_GLIBCXX_DEBUG -o nain 
S ./nain 
1 

/opt/gcc_latest/include/c++/7.0.0/debug/safe_iterator.h:270: 

Error: attenpt to dereference a singular iterator. 

Objects involved in the operation: 

iterator "this" @ 0x0x7ffc06323730 { 

type = _gnu_debug::_Safe_iterator<_gnu_cxx::_nornal_iterator<int*, std::_cxxl998::vecto 

r<int, std::allocator<int> > >, std::_debug::vector<int, std::allocator<int> > > (nutable iterat 

or); 

state = singular; 

references sequence with type 'std::_debug::vector<int, std::allocator<int> >' @ 0x0x7ffc0 

6323760 

} 

aborted (core dunped) _ 


8. The LLVM/clang implementation of the STL also has debug flags, but 
they serve the purpose of debugging the STL itself, not user code. For 

user code, you can activate different sanitizers. Let's compile the code 

/\ 

with clang using the A -fsanitize=address - 
f sanitize=undef inedA flags and see what happens: 




S clang++ -std=c++lz -fsanitize=address -fsanitize=undefined nain.cpp -o main 
$ ./nain 
1 


==20639==ERR0R: AddressSanitizer: heap-use-after-free on address Ox6O2OOOO0effO at pc 9x0000004eb 
519 bp 0x7ffc0fe5d730 sp Ox7ffc0fe5d728 
READ of size 4 at Ox6020000OeffO thread T0 

#0 0x4eb518 (/hone/tfc/src/stl_cookbook/03/checked_iterator/piain+0x4eb518) 

#1 0x7fIc89f893f0 (/Iib/x86_64-iinux-gnu/iibc.so.6+0x203f0) 

#2 0x4187f9 (/home/tfc/src/stl_cookbook/03/checked_iterator/nain+0x4187f9) 

0x60200000effO is located 0 bytes inside of 12-byte region [0x6020O000eff0,0x60200000effc) 
freed by thread TO here: 

#0 0x4e83c0 (/hopie/tfc/src/stl_cookbook/03/checked_iterator/piain+Ox4e83c0) 

#1 0x4ee64b (/home/tfc/src/stl_cookbook/03/checked_iterator/piain+0x4ee64b) 

#2 0x4ee619 (/hone/tfc/src/stl_cookbook/03/checked_iterator/nain+Ox4ee619) 

#3 0x4ee4ae (/hone/tfc/src/stl_cookbook/03/checked_iterator/nain+0x4ee4ae) 

#4 0x4f09c7 (/home/tfc/src/stl_cookbook/03/checked_iterator/piain+0x4f09c7) 

#5 0x4ef2a7 (/horie/tfc/src/stl_cookbook/03/checked_iterator/nain+0x4ef2a7) 

#6 0x4eclaf (/hone/tfc/src/stl_cookbook/03/checked_iterator/riain+0x4eclaf) 

#7 0x4eb364 (/hone/tfc/src/stl_cookbook/03/checked_iterator/main+0x4eb364) 

#8 0x7fIc89f893f0 (/lib/x86_64-iinux-gnu/libc.so.6+0x203f0) 

previously allocated by thread TO here: 

#0 0x4e7dc0 (/horie/tfc/src/stl_cookbook/03/checked_iterator/main+0x4e7dc0) 


Wow , this is a very precise description of what went wrong. The screenshot 

/\ /V 

would have spannedA multiple pages of this book if it hadA not been 
truncated. Note that this is not a clang-only feature, as it also works with 


GCC. 


Note 

If you get runtime errors because some library is missing, then 
your compiler did not automatically ship with libasan and 
libubsan. Try to install them via your package manager or 
something similar. 

How it works... 

As we have seen, we did not need to change anything in the code in order to 
get this kind of tripwire feature for buggy code. It basically came for free, 
just by appending some compiler flags to the command line when compiling 
the program. 

This feature is implemented by sanitizers. A sanitizer usually consists of an 

/V 

additional compiler moduleA and a runtime library. When sanitizers are 
activated, the compiler will add additionalinformation and code to the binary, 

/V 

which results from our program. At runtime, the sanitizer libraries thatA are 





/\ /\ 

then linked into the program binary can, for example,A replaceA the 

/\ /V 

maiiocA and free functions in order to analyze A how the program deals 
with the memory it acquires. 

Sanitizers can detect different kinds of bugs. Just to list a fewA valuable 
examples: 

/\ 

• Out-of-bounds: A This triggers whenever we access an array, vector, or 
anything similar outside its legitimate memory range. 

• Use-after-free: This triggers if we reference heap memory after it was 
already freed (which we did in this section). 

• Integer overflow: This triggers if an integer variable overflows by 

/V 

calculating with values that A do not fit into the variable. For signed 
integers, the arithmetic wraparound is undefined behavior. 

/V 

• Pointer alignment: Some architectures cannot access memoryA if it has 
a weird alignment in memory. 

There are many more such bugs that sanitizers can detect. 

It is not feasible to always activate all available sanitizers because they make 
the program slower. However, it is good style to always activate sanitizers in 
your unit tests and integration tests. 

There’s more... 

There are a lot of different sanitizers for different bug categories, and they are 
all still under development. We can and should inform ourselves on the 
internet about how we can improve our test binaries. The GCC and LLVM 
project homepages list their sanitizing capabilities in their online 
documentation pages: 

• https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html 

/\ /\ /\ 

• http: //clang. llvm. or g/docs/index. html A (look for A sanitizers A in the 
table of contents) 

* 

Thorough testing with sanitizers is something thatA every programmer 

/V /V 

should be aware of and shouldA always A be doing. Unfortunately, this is not 




the case in alarmingly many companies, although buggy code is the 

/\ /\ 

mostA importantA entry point for all the malware and computer viruses out 

/V 

there. A 

/V 

When you get a new job as a software developer, check if your teamA really 
uses all the sanitizing possibilities there are. If not, you have the unique 
chance to fix important and sneaky bugs on your first day at work! 



Building your own zip iterator adapter 


Different programming languages lead to different programming styles. This 
is, because there are different ways to express things, and they are differing in 

_ /V 

their elegance for each use case. That is no surpriseA because every language 
was designed with specific objectives. 

A very special kind of programming style is purelyfunctionalprogramming. 

/V 

It is magically different from the imperative programming which CA or C++ 
programmers are used to. While this style is very different, it enables 
extremely elegant code in many situations. 

/V 

One example ofA this elegance is the implementation of formulas, such 

/V 

asA the mathematical dot product. Given two mathematical vectors, applying 
the dot product to them means pairwise multiplying of the numbers at the 
same positions in the vector and then summing up all of those multiplied 
values. The dot product of two vectors (a, b, c) * (d, e, f)is(a*e + 
b * e + c * f) . Of course, we can do that with C and C++, too. It could 
look like the following: 


std::vector<double> a {1.0, 2.0, 3.0}; 
std::vector<double> b {4.0, 5.0, 6.0}; 

double sum {0}; 

for (size_t i {0}; i < a.size(); ++i) { 

sum += a[i] * b[i]; 

} 

// sum = 32.0 

/v 

How does it look like in those languages thatA can be considered more 
elegant ? 

Haskell is a purely functional language, and this is how you can calculate the 
dot product of two vectors with a magical one-liner: 



[Prelude> a = [1.0, 2.0, 3.0] 
[Prelude> b = [4.0, 5.0, 6.0] 
[Prelude> sum $ zipWith (*) a b 
32.0 


Python is not a purely functional language, but it supports similar patterns to 
some extent, as seen in the next example: 


[>» a = [1.0, 2.0, 3.0] 

[>» b = [4.0, 5.0, 6.0] 

[>» sum([ p [0] * p[l] for p in zip(a, b) ]) 
32.0 


The STL provides a specific algorithm called std:: inner product, which 
solves this specific problem in one line, too. But the point is that in 

/V 

manyA other languages, such code can be written on the fly in only one line 
without specific library functions that support that exact purpose. 

Without delving into the explanations of such foreign syntax, an important 

/\ 

commonality inA both examples is the magical zip function. What does it 
do? It takes the two vectors a and b and transforms them to a mixed vector. 
Example: [al, a2, a3] and [bl, b2, b3] result in [ (al, bl), (a2, 
b 2 ), (a3, b3 ) ] when they are zipped together. Have a close look at it; it's 
really similar to how zip fasteners work! 

The relevant point is that it is now possible to iterate over one A combined 
range where pairwise multiplications can be done and then summed up to an 
accumulator variable. Exactly the same happens in the Haskell and Python 
examples, without adding any loop or index variable noise. 

It will not be possible to make the C++ code exactly as elegant and generic as 
in Haskell or Python, but this section explains how to implement similar 
magic using iterators, by implementing a zip iterator. The example problem 
of calculating the dot product of two vectors is solved more elegantly by 
specific libraries, which are beyond the scope of this book. However, this 
section tries to show how much iterator-based libraries can help in writing 
expressive code by providing extremely generic building blocks. 


How to do it... 





In this section, we will recreate the zip function as known from Haskell or 
Python. It will be hardcoded to vectors of double variables in order to not 
distract from iterator mechanics: 

1. First, we need to include some headers: 

#include <iostream> 

#include <vector> 

#include <numeric> 

2. Next, we define the zip iterator class. While iterating over a 
zip iterator range, we will get a pair of values from the two 
containers at every iteration step. This means that we iterate over two 
containers at the same time: 

class zip_iterator { 

3. The zip iterator needs to save two iterators, one for each container: 

using it_type = std::vector<double>::iterator; 

it_type itl; 
it_type it2; 


4. The constructor simply saves the iterators from the two containers that 
we would like to iterate over: 


public: 

zip iterator(it_type iteratorl, it_type iterator2) 

: it1{iteratorl}, it2{iterator2} 

{ } 

5. Incrementing the zip iterator means incrementing both the member 
iterators: 


zip_iterator& operator++() { 
++i 11; 

++it2; 

return *this; 


} 



_ A 

6. Two zip iterators are unequal A if both the member iterators are unequal 
to their counterparts in the other zip iterator. Usually, one would use 

logical or (| |) instead of and (&&), but imagine that the ranges are not of 

/\ 

equal length. In such aA case, it would not be possible to match 

/V 

both A. the end iterators at the same time. This way, we can abort the 
loop when we reach the first end iterator of either range: 

bool operator!=(const zip_iterator& o) const { 
return itl != o.itl && it2 != o.it2; 

} 

7. The equality comparison operator is just implemented using the other 
operator, but negating the result: 


bool operator==(const zip_iterator& o) const { 
return !operator!=(o); 

} 

8. Dereferencing the zip iterator gives access to the elements of both the 
containers at the same position: 


std::pair<double, double> operator* () const { 
return {*itl, *it2}; 

} 


9. This was the iterator code. We need to make the iterator compatible with 
STL algorithms, so we define the needed type trait boilerplate code for 
that. It basically says that this iterator is just a forward iterator, and it 
returns pairs of double values when dereferenced. Although we do not 
use difference type in this recipe, different implementations of the 
STL might need it in order to compile: 


namespace std { 
template <> 

struct iterator_traits<zip_iterator> { 

using iterator_category = std::forward_iterator_tag; 
using value_type = std: :pair<double, doublet- 
using difference_type = long int; 



} 


_ A 

10. The next step is to define a range class thatA returns us zip iterators 
from its begin and end functions: 

class zipper { 

using vec_type = std::vector<double>; 
vec_type &vecl; 
vec_type &vec2; 


11. It needs to reference two existing containers in order to form zip 
iterators from them: 


public: 

zipper(vec_type &va, vec_type &vb) 

: vecl{va}, vec2{vb} 

{ } 

12. The begin and end functions just feed pairs of begin and end pointers in 
order to construct zip iterator instances from that: 

zip^iterator begin() const { 

return {std::begin(vecl), std::begin(vec2)}; 

} 

zip^iterator end() const { 

return {std::end(vecl), std::end(vec2)}; 

} 

}; 

13. Just as in the Haskell and Python examples, we define two vectors of 
double values. We also define that we use namespace std within the 
main function by default: 


int main() 

{ 

using namespace std; 
vector<double> a {1.0, 2.0, 3.0}; 
vector<double> b {4.0, 5.0, 6.0}; 

14. The zipper object combines them to one vector-like range where we see 
pairs of a and b values: 



zipper zipped {a, b}; 


15. We will use std:: accumulate in order to sum all the items of the range 
together. We can't do it directlyA because that would mean that we sum 
up the instances of std: :pair<doubie, doubie> for which the concept 
of sum is not defined. Therefore, we will define a helper lambda 

/V 

thatA takes a pair, multiplies its members, and adds it to an accumulator. 
TheA std:: accumulate works well with lambdas with such a signature; 

const auto add_product ([](double sum, const auto &p) { 

return sum + p.first * p.second; 

}) ; 

16. Now, we feed it to std:: accumulate, together with the begin and end 
iterator pair of the zipped ranges and a start value of o. o for the 
accumulator variable, which, in the end, contains the sum of the 
products: 

const auto dot_product (accumulate ( 

begin(zipped), end(zipped), 0.0, add_product)); 

17. Let's print the dot product result:A 

cout << dot_product << 1 \n 1 ; 

} 

18. Compiling and running the program yields the correct result: 

32 

There’s more... 

OK, that was a lot of work for a little bit of syntax sugar, and it's still not as 

elegant as Haskell code can be without any effort. A big flaw is the 

/\ 

hardcoded nature of our little zip iterator—it only works on theA std:: vector 
ranges of double variables. With a bit of template code and some type traits, 
the zipper can be made more generic. This way, it could combine lists and 
vectors, or deques and maps, even if these are specialized on completely 



different container item types. 


The amount of work and thought needed in order to really and correctly make 
such classes generic is not to be underestimated. Luckily, such libraries do 
already exist. One popular non-STL library is the 5oo,s/zip_iterator. It is 

/V 

very generic A and easy to use. 

By the way, if you came here to see the most elegant way to do a dot product 
in C++, and don't really care about the concept of zip-iterators, you should 
have a look at std:: vaiarray. See for yourself: 

#include <iostream> 

#include <valarray> 

int main() 

{ 

std::valarray<double> a {1.0, 2.0, 3.0}; 
std::valarray<double> b {4.0, 5.0, 6.0}; 

std::cout << (a * b).sum() << 1 \n 1 ; 

} 


Ranges library 

There is a very, very interesting C++ library, which supports zippers and all 
other kinds of magic iterator adapters, filters, and so on: the ranges library. It 
is inspired by the Boost ranges library, and for some time, it looked like it 
would find its way into C++17, but unfortunately, we will have to wait for 
the next standard. The reason why this is so unfortunateA is that it will 

/\ /V 

vigorously improve the possibilities ofA writingA expressive and fast code in 

/\ 

C++ by composing complex functionality from generic A and simple blocks 
of code. 

There are some very simple examples in its documentation: 

1. Calculating the sum of the squares of all numbers from l to 10 : 


const int sum = accumulate(view::ints (1) 

| view::transform([](int i){return 
| view::take (10), 0); 



2. Filtering out all uneven numbers from a numeric vector, and 
transforming the rest to strings: 

std::vector<int> v {1,2,3,4,5,6,7,8,9,10}; 

auto rng = v | view::remove_if([](int i){return i % 2 == 1; 

| view::transform([](int i){return std::to_str 

// rng == {"2"s,"4"s,"6"s,"8"s,"10"s } ; 

If you are interested and can't wait for the next C++ standard, have a look at 

/y /y 

the ranges documentationA atA https://ericniebler.github.io/range-v3/ . 



Chapter 4. Lambda Expressions 

/\ 

WeA will cover the following recipes in this chapter: 

• Defining functions on the run using lambda expressions 

• Adding polymorphy by wrapping lambdas into std:: function 

• Composing functions by concatenation 

• Creating complex predicates with logical conjunction 

• Calling multiple functions with the same input 

• Implementing transform if using std:: accumulate and lambdas 

• Generating cartesian product pairs of any input at compile time 



Introduction 


/V 

One important new feature of C++11 wasA lambda expressions. In C++14 

/V 

and C++17, the lambda expressionsA got some new additions, which have 

/\ 

made them even more powerful. But first,A what is a lambda expression? 

/V 

Lambda expressionsA or lambda functions construct closures. A closure is a 

/V /V 

very generic termA for unnamed objects A. that can be called like functions. 

/\ 

In order to provide such a capabilityA in C++, such an object must 
implement the () function calling operator, with or without parameters. 
Constructing such an object without lambda expressions before C++11 could 
still look like the following: 

#include <iostream> 

#include <string> 

int main() { 

struct name_greeter { 
std::string name; 

void operator () () { 

std::cout << "Hello, " << name << '\n 1 ; 

} 

}; 

name_greeter greet_john_doe {"John Doe"}; 
greet_john_doe(); 

} 

/\ /v 

Instances of theA name greeterA struct obviously carry a string with them. 
Note that both this structure type and instance are not unnamed butA lambda 
expressions can be, as we will see. In terms of closures, we would say they 
capture a string. When the example instance is called like a function without 

/V 

parameters, it prints "Hello, John Doe "A because we constructed it with 
this name. 

Since C++11, it has become easier to create such closures: 




#include <iostream> 


int main() { 

auto greet_john_doe ([] { 

std::cout << "Hello, John Doe\n"; 

}) ; 

greet_john_doe(); 

} 

/v /v 

That's it. The whole struct,A name greeter, isA replaced by a little [] { /* 
do something */ } construct, which might look a bit like magic at first, but 

/v /\ 

the first section A of this chapter will explain it A thoroughly in all the 
possible variants. 

Lambda expressions are of a great help to make code generic and 

/\ _ /V 

tidy.A They can be used as parameters for very generic algorithmsA in order 

/V 

to specialize what those A do when processing specific user-defined types. 
They can also be used to wrap work packages together with data in order to 

/V 

be run in threadsA or just to save work and postpone the actual execution. 
Since C++11 came out, more and more libraries work with lambda 
expressions because they became a very natural thing in C++. Another use 

/V 

case is metaprogramming, becauseA lambda expressions can also be 
evaluated at compile time. However, we are not going much into that 
direction, as this would quickly blast the scope of this book. 

This chapter does heavily rely on some functional programming patterns, 
which might look weird to novices or programmers who are already 
experienced but not with such patterns. If you see lambda expressions in the 

/V 

coming recipes that return lambda expressions, whichA again return lambda 
expressions, please don't feel frustrated or confused too quickly. We are 
pushing the boundaries a bit in order to prepare ourselves for modern C++, 
where functional programming patterns occur with increasing regularity. If 
some code in the following recipes looks a bit too complex, take your time to 
understand it. Once you got through this, complex lambda expressions in real 
projects in the wild will not confuse you any longer. 



Defining functions on the run using lambda 
expressions 


With lambda expressions, we can encapsulate code in order to call it later, 
and that also might be somewhere else because we can copy them around. 

We can also just encapsulate code to call it multiple times with slightly 
different parameters without having to implement a whole new function class 
for that task. 

The syntax of lambda expressions was really new in C++11, and it has 

/V 

slightly evolvedA with the next two standard versions until C++17. In this 
section, we will see what lambda expressions can look like and what they 
mean. 


How to do it... 

We are going to write a little program in which we play with lambda 
expressions in order to get a feeling for them: 

1. Lambda expressions do not need any library support, but we are going to 
write messages to the terminal and use strings, so we need the headers 
for this: 

#include <iostream> 

#include <string> 


2. Everything happens in the main function this time. We define two 

/\ 

function objects that take no parameters A and return integer constants 

/V 

with the values,A l and 2. Note that the return statement is surrounded 

/V 

by curly bracketsA {}, like it is in normal functions, and the () 
parentheses, which denote a parameterless function, are optional, A we 
don't provide them in the second lambda expression. But the [ ] brackets 



have to be there: 


int main() 

{ 

auto just_one ( [](){ return 1; } ); 
auto just_two ( [] { return 2; } ); 

3. Now, we can call both the function objects just by writing the names of 
the variables they are saved to and appending the parentheses. In this 
single line, they are indistinguishable from normal functions for the 
reader: 


std::cout << just_one() << ", " << just_two() << '\n'; 

4. Now let's forget about those and define another function object, which is 

/V /\ 

called plus A because it takes two parameters and returns theirA sum: 

auto plus ( [](auto 1, auto r) { return 1 + r; } ); 

/v 

5. This is also easy to use, just like any other binary function. AsA we 

defined its parameters to be of the auto type, it will work with anything 

/\ 

thatA defines the plus operator +, just as strings do: 

std::cout << plus(l, 2) << '\n'; 

std::cout << plus(std::string!"a"}, "b") << '\n'; 


6. We do not need to store a lambda expression in a variable in order to use 

/V 

it. We can also define it in place and then write A the parameters in 
parentheses just behind it (l, 2 ): 


std::cout 

<< [] (auto 1, auto r) { return 1 + r; } (1, 2) 
« ' \ n ' ; 


/v 

7. Next, we will define a closure thatA carries an integer counter value 

/V 

around with it. Whenever we call it, it increments its counter valueA and 
returns the new value. In order to tell it that it has an internal counter 
variable, we write count = o within the brackets to tell it that there is a 

/\ /V 

variable count A initialized to the integer valueA o. In order to allow it 



to modify its own captured variables, we use the mutable keyword, as 
the compiler would not allow it otherwise: 


auto counter ( 

[count =0] () mutable { return ++count; } 

) ; 


8. Now, let's call the function object five times and print the values it 
returns, so we can see the increasing number values later: 

for (size_t i {0}; i < 5; ++i) { 

std::cout << counter() << ", 

} 

std::cout << 1 \n 1 ; 

9. We can also take existing variables and capture them by reference 
instead of giving a closure its own value copy. This way, the captured 

variableA can be incremented by the closure, but it is still accessible 

/\ /\ 

outside. In order to do so, we writeA &aA between the brackets, where 
the & means that we store only a reference to the variable, not a copy. 

int a {0} ; 

auto incrementer ( [& a] { ++a; } ); 

10. If this works, then we should be able to call this function object multiple 
times, and then observe that it has really changed the value of variable a: 

incrementer(); 
incrementer(); 
incrementer(); 

std::cout 

<< "Value of 'a' after 3 incrementer() calls: " 

<< a << '\n'; 


11. The last example is currying. Currying means that we take a 

/\ 

functionA that can accept some parameters and store it in another 

/V 

function object, which accepts A fewer parameters. In this case, we store 

/V 

the plus functionA and only accept one parameter, which we forward to 
the plus function. The other parameter is the value 10, which we save in 



_ /\ 

the function object. This way, we getA a function, which 

/\ /V /V /\ 

weA callA piusten because it can addA that value to the A single 
parameter it accepts: 

auto plus_ten ( [ = ] (int x) { return plus(10, x) ; } ) ; 

std::cout << plus_ten(5) << '\n'; 

} 

12. Before compiling and running the program, go through the code again 
and try to foresee what it will print to the terminal. Then run it and 
check against the real output: 

1, 2 
3 

ab 

3 

1, 2, 3, 4, 5, 

Value of a after 3 incrementer() calls: 3 
15 

How it works... 

What we just did was not overly complicated—we added numbers, and 

incremented and printed them. We even concatenated strings with a function 

/\ 

object, which was implemented to add up A numbers. But for anyone who 
didn't know lambda expression syntax yet, it might have looked confusing. 

So, let's first have a look at all the lambda expression peculiarities: 



[capture list] (parameters) 

mutable (optional) 

constexpr (optional) 

exception attr (optional) 

-> return type (optional) 

{ 

body 

} 


We can usually omit most of this, which spares us some typing, in the 
average case. The shortest lambda expression possible is [ ] {}. It accepts no 
parameters, captures nothing, and essentially does nothing. 

So what does the rest mean? 

Capture list 

Specifies if and what we capture. There are several forms to do so. There are 
two lazy variants: 

• If we write [ = ] () {...}, we capture every variable the closure 
references from outside by value, which means that the values are 
copied 

• Writing [ & ] () {...} means that everything the closure references 
outside is only captured by reference , which does not lead to a copy. 

Of course, we can set the capturing settings for every variable individually. 

/V 

Writing [a, &b] () {...} means, that we capture the variableA a by value, 
and b by reference. This is more typing work, but it's generally safer to be 

/V 

that verboseA because we cannot accidentally capture something we don't 



want to capture from outside. 


In the recipe, we defined a lambda expression as such: [ count=o ] () 

In this special case, we did not capture any variable from outside, but we 
defined a new one called count. Its type is deduced from the value we 
initialized it with, namely o, so it's an in t. 

It is also possible to capture some variables by value and some, by reference, 
as in: 


• [a, &b] o {...}: This captures a by copy and b by reference. 

• [&, a] o {...}: This captures a by copy and any other used variable 
by reference. 

• [=, &b, i {22}, this] o {...}: This captures b by reference, this 
by copy, initializes a new variable i with value 22, and captures any 
other used variable by copy. 

Note 

If you try to capture a member variable of an object, you cannot 
do this directly using [member a] 0 {...}. Instead, you have to 
capture either this or *this. 


mutable (optional) 

If the function object should be able to modify the variables it captures by 
copy ([ = ]), it must be defined mutable. This includes calling non-const 
methods of captured objects. 

constexpr (optional) 

If we mark the lambda expression explicitly as constexpr, the compiler will 
error out if it does not satisfy the criteria of constexpr functions. The 

/V /V 

advantage ofA constexprA functions and lambda expressions is that the 



compiler can evaluate their result at compile time if they are called with 
compile-time constant parameters. This leads to less code in the binary later. 

/V 

If we do not explicitly declare the lambda expression to be constexprA but it 

/V 

fits the requirements for that, it will be implicitlyA constexpr anyway. If we 
want a lambda expression to be constexpr, it helps to be explicit because the 
compiler will then help us by erroring out if we did it wrong. 

exception attr (optional) 

This is the place to specify ifA the function object can throw 

/\ 

exceptionsA when it's called and runs into an error case. 

return type (optional) 

If we want to have ultimate control over the return type, we may not want the 

/V 

compiler to deduce it for us automatically. In such aA case, we can just write 
[] o -> Foo {}, which tells the compiler that we will really always return 

/V /V 

theA FooA type. 



Adding polymorphy by wrapping lambdas 
into std::function 


/V 

Let's say we want to write anA observer function for some kind of value, 
which might change sometimes, which then notifies other objects; like a gas 
pressure indicator, or a stock price, or something similar. Whenever the value 
changes, a list of observer objects should be called, which then react their 
way. 

In order to implement this, we could store a range of observer function 
objects in a vector, which all accept an int variable as the parameter, which 

represents the observed value. We do not know what these function objects 

/\ 

do in particularA when they are called with the new value, but we also don't 
care. 

Of what type will that vector of function objects be? 

/V /V /V 

TheA std:: vector<void (*) (int) >A type would be correctA if we were 

/\ /\ 

capturing pointers toA functions with signatures such asA void f (int);. 

This would indeed also work with any lambda expression thatA does not 

/V 

capture any variables, such asA [ ] (int x) {...}. But a lambda expression 

/V /\ 

thatA captures somethingA is actually a completely different type compared 
/\ /\ 

withA a normal functionA because it's not just a function pointer. It is an 

/\ _ /\ 

objectA that couples a certain amount of data with a function! Think ofA pre- 
C++11 times, when there were no lambdas. Classes and structs are the natural 
way of coupling data with functions, and if you change the data member 
types of a class, you get a completely different class type. It's just natural that 
a vector can't store completely different types using the same type name. 

Telling the user that it's only possible to save observer function objects 

/V /\ 

thatA do not capture anything is badA because it limits the number of use 
cases very much. How can we allow the user to store any kind of function 
object, only constraining to the call interface, which takes a specific set of 
parameters that represent the value that shall be observed? 




This section shows how to solve this problem using std:: function, which 
can act as a polymorphic wrapper around any lambda expression, no matter if 
and what it captures. 


How to do it... 


/\ 

In this section, we are going to create several lambda expressions thatA are 

/\ /\ 

completely different in regard toA the variable types they captureA but have 
the same function call signature in common. These will be saved in one 
vector using std: : function: 


1. Let's first do some necessary includes: 

#include <iostream> 

#include <deque> 

#include <list> 

#include <vector> 

#include <functional> 


/v 

2. We implement a little function that returns aA lambda expression. It 

/V 

accepts a container and returns a function object thatA captures that 
container by reference. The function object itself accepts an integer 
parameter. Whenever that function object is fed with an integer, it will 
append that integer to the container it captures: 

static auto consumer (auto Scontainer){ 
return [&] (auto value) { 

container.push_back(value); 

}; 

} 

3. Another little helper function will print whatever container instance we 
provide as a parameter: 


static void print (const auto &c) 
{ 

for (auto i : c) { 

std::cout << i << ", 

} 



std::cout << '\n'; 

} 

/v 

4. In the main function, we first A instantiate adeque,aiist, and a 
vector, which all store integers: 


int main() 

{ 

std::deque<int> d; 
std::list<int> 1; 
std::vector<int> v; 


/V /V /V 

5. Now we useA theA consumer function with our container instances,A d, 
l, and v: we produce consumer function objects for those and store them 
all in a vector instance. ThenA we have a vector thatA stores 
threeA function objects. These function objects each capture a reference 
to one of the container objects. These container objects are of 
completely different types and so are the function objects. Nevertheless, 

the vector holds instances of std:: function<void (int) >. All the 

/\ 

function objects are implicitly wrapped intoA such std:: function 
objects, which are then stored in the vector: 


const std::vector<std::function<void(int)>> consumers 
{consumer(d), consumer(1), consumer(v)}; 


6. Now, we feed 10 integer values to all the data structures by looping over 

/V 

the values A and then looping over the consumer function objects, which 
we call with those values: 

for (size_t i {0}; i < 10; ++i) { 

for (auto &&consume : consumers) { 
consume(i); 

} 

} 

/v 

7. All the threeA containers should now contain the same 10 number 

/V 

values.A Let's print their content: 


print(d); 
print(1); 




print(v); 

} 

8. Compiling and running the program yields the following output, which 
is just what we expect: 


$ ./std function 


0 , 

A 

2, 

3, 

4, 

5, 

6, 

7, 

00 

9, 

0 , 

A 

2, 

3, 

4, 

5, 

6, 

7, 

8, 

9, 

0 , 

A 

2, 

3, 

4, 

5, 

6, 

7, 

8, 

9, 


How it works... 

The complicated part of this recipe is the following line: 

const std::vector<std::function<void(int)>> consumers 
{consumer(d), consumer(1), consumer (v) A* 

The objects d, 1 , and v are each wrapped into a consumer (...) call. This call 

/V 

returns function objects, which then each capture references to one ofA d, 1 , 

/\ 

andA v. Although these function objects all accept int values as parameters, 

the fact that they capture completely different variables also makes them 

_ /\ 

completely different types. This is like trying to stuff variables of typeA a, b, 
and c into a vector, although these types have nothing in common. 

In order to fix this, we need to find a common type, which can store very 

/V 

different function objects, that is,A std: : function. An 

std:: function<void (int) > object can store any function object or 

/\ 

traditionalA function, which accepts an integer parameter and returns 
nothing. It decouples its type from the underlying function object type, using 

/V 

polymorphy. Consider A we write something like this: 

std::function<void(int)> f ( 

[Svector](int x) { vector.push_back(x); }); 

/v 

Here,A the function object which is constructed from the lambda 

/V 

expressionA is wrapped into an std:: function object, and whenever we call 
f (123) , this leads to a virtual function call, which is redirected to the actual 
function object inside it. 



While storing function objects, std:: function instances apply some 

/\ 

intelligence. A If we capture more and more variables in a lambda expression, 
it must grow larger. If its size is not too large, std:: function can store it 
within itself. If the size of the stored function object is too large, 
std:: function will allocate a chunk of memory on the heap and then store 

the large function object there. This does not affect the functionality of our 

/\ 

code, but we should know about thisA because this can impact the 
performance of our code. 

Note 

A lot of novice programmers think or hope that 
std:: function< ... > actually expresses the type of a lambda 
expression. No, it doesn't. It is a polymorphic library helper, 
which is useful for wrapping lambda expressions and erasing 
their type differences. 



Composing functions by concatenation 


-I 


A lot of tasks are not really worthy of being implemented in completely 
custom code. Let's, for example, have a look on how a programmer might 
solve the task of finding out how many unique words a text contains with the 
programming language Haskell. The first line defines a 

/V 

functionA unique words and the second one demonstrates its use with an 
example string: 


Prelude> unique_words = length . group . sort . words . (map toLower) 
Prelude> unique_words "A B c d a b c d e" 

5 


Wow, that is short! Without explaining Haskell syntax too much, let's see 
what the code does. It defines the function called unique words, which 
applies a series of functions to its input. It first maps all the characters from 

/y _ /v a 

the input to lowercase withA map toLower. This way, words like A fooA and 
f oo can be regarded as the same word. Then, theA words function splits a 

/V 

sentence into individual words, asA from "foo bar baz" to ["foo", "bar", 
"baz" ]. Next step is sorting the new list of words. This way, a word sequence 
suchasAfa", "b", "a"] becomes ["a", "a", "b"] . Now, theA group 

a /y 

function takes over. It groupsA consecutive equal words into groupedA lists, 
so ["a", "a", "b"] becomes [ ["a", "a"], ["b"] ]. The job is now 

/v 

nearly done, as we now only need to count how many A groups of equal 
words we got, which is exactly what the length function does. 

This is a wonderful style of programming, as we can read what happens from 
right to leftA because we are just, kind of, describing a transformation 
pipeline. We don't need to care how the individual pieces are implemented 
(unless it turns out that they are slow or buggy). 

A. 

However, we are not here to praise Haskell A but to improve our C++ skills. 

It is possible to work like this in C++ too. We will not completely reach the 
elegance of the Haskell example but we still have the fastest programming 






language there is. This example explains how to imitate function 
concatenation in C++ with lambda expressions. 


How to do it... 


In this section, we define some simple toy function objects and concatenate 

/V 

them, so we get a single function thatA applies the simple toy functions after 
each other to the input we give it. In order to do so, we write our own 
concatenation helper function: 

1. First, we need some includes: 

#include <iostream> 

#include <functional> 

/v 

2. Then, we implement the helper function,A concat, which arbitrarily 
takes many parameters. These parameters will be functions, such asA f, 
g, and h, and the result will be another function object thatA applies 

f (g (h (...))) on any input: 

template <typename T, typename ...Ts> 
auto concat(T t, Ts ...ts) 

{ 


3. Now, it gets a little complicated. When the user provides functions f, g, 

/\ 

and h, we will evaluate this toA f ( concat (g, h) ), which again 
expands to f( g( concat (h) ) ), where the recursion aborts, so we get 

/V 

f ( g ( h (...) ) ). This chain of function calls representingA the 

/V 

concatenation of these user functionsA is captured by a lambda 

/\ 

expression, which can later take some parameters,A p, and then forward 

/\ 

them to A f (g (h (p))) . This lambda expression is what we return. The 

/V 

if constexpr construct checks whetherA we are in a recursion step 
with more than one function to concatenate left: 

if constexpr (sizeof...(ts) > 0) { 

return [=](auto ...parameters) { 

return t (concat(ts...) (parameters...)); 



4. The other branch of the if constexpr construct is selected by the 

/V /V 

compilerA if we are at the end of the recursion. In such aA case, we just 

/V 

return the function,A t, because it is the only parameter left: 

else { 

return t; 

} 

} 

5. Now, let's use our cool new function concatenation helper with some 
functions we want to see concatenated. Let's begin with the main 
function, where we define two cheap simple function objects: 


int main() 

{ 

auto twice ([] (int i) { return i * 2; }); 
auto thrice ([] (int i) { return i * 3; }); 

/v 

6. Now let's concatenate.A We concatenate our two multiplier function 
objects with the STL function,A std: :pius<int>, which takes two 
parameters and simply returns their sum. This way, we get a function 

/V 

thatA does twice(thrice(plus( a, b ))). 
auto combined ( 

concat(twice, thrice, std::plus<int>{}) 

) ; 


/\ /\ 

7. Now let's use it. TheA combinedA function looks like a single normal 
function now, and the compiler is also able to concatenate those 
functions without any unnecessary overhead: 

std::cout << combined (2, 3) << 1 \n 1 ; 

} 

8. Compiling and running our program yields the following output, which 
we also expected, because 2 * 3 * (2 + 3)is3o: 



$ ./concatenation 
30 


How it works... 


The complicated thing in this section is the concatA function. It looks 
horribly complicated because it unpacks the parameter pack ts into another 
lambda expression, which recursively calls concat again, with less 
parameters: 

template <typename T, typename ...Ts> 
auto concat(T t, Ts ...ts) 

{ 

if constexpr (sizeof...(ts) > 0) { 

return [=](auto ...parameters) { 

return t(concat(ts... ) (parameters...)); 

}; 

} else { 

return [=](auto ...parameters) { 
return t(parameters...); 

}; 

} 

} 

Let's write a simpler version, which concatenates exactly three functions: 

template <typename F, typename G, typename H> 
auto concat (F f, G g, H h) 

{ 

return [=](auto ... params) { 

return f( g( h( params... ) ) ); 

}; 

} 

This already looks similar, but less complicated. We return a lambda 

expression, which captures f, g, and h. This lambda expression arbitrarily 

/\ 

accepts many parametersA and just forwards them to a call chain of f, g, and 
h. When we write auto combined (concat (f, g, h)), and later call that 

/V 

function object with two parameters, such asA combined ( 2 , 3 ), then the 2 , 

/V /\ /V /\ 

3 A are represented byA the paramsA pack from the precedingA concat 
function. 



/V 

Looking at the much more complex, generic concat function again;A the 
only thing we do really differently is the f ( g ( h ( params ...))) 
concatenation. Instead, we write f ( concat (g, h) ) (params...), which 
evaluates to f ( g ( concat (h) ) ) (params.. .) in the next recursive call, 
which then finallyA results in f ( g ( h ( params... ) ) ). 



Creating complex predicates with logical 
conjunction 


When filtering data with generic code, we end up defining predicates, which 
tell what data we want, and what data we do not want. Sometimes predicates 

/V /V 

are theA combinations of differentA predicates. 

When filtering strings, for example, we could implement a predicate 

/V /V 

thatA returns trueA if its input string begins with "foo". Another predicate 

/V 

could return trueA if its input string ends with "bar". 

Instead of writing custom predicates all the time, we can reuse predicates by 

/\ /\ 

combining them. If we want to filter strings thatA begin with "foo "A and end 
with "bar", we can just pick our existing predicates and combine them with a 
logical and. In this section, we play with lambda expressions in order to find 
a comfortable way to do this. 

How to do it... 


We will implement very simple string filter predicates, and then we will 

/\ 

combine them with a little helper function thatA does the combination for us 
in a generic way. 

1. As always, we'll include some headers first: 

#include <iostream> 

#include <functional> 

#include <string> 

#include <iterator> 

#include <algorithm> 

/v 

2. Because we are going to need them later, we implement twoA simple 

_ /V 

predicate functions. The first one tells if a string begins A with the 
character ' a' and the second one tells if a string ends with the character 




' b': 


static bool begins_with_a (const std::string &s) 

{ 

return s.find("a") == 0; 

} 

static bool ends_with_b (const std::string &s) 

{ 

return s.rfind("b") == s.length() - 1; 

} 

/\ /\ /v 

3. Now, let's implement a helper function, whichA weA callA combine. It 
takes a binary function as itsA first parameter, which could be 

/V /V 

theA logical andA function or the logical or function, for example. 
Then, it takes two other parameters, which shall be two predicate 

/V 

functions that A are then combined: 

template <typename A, typename B, typename F> 
auto combine(F binary_func, A a, B b) 

{ 

/v 

4. We simply return a lambda expression thatA captures the new predicate 
combination. It forwards a parameter to both predicates and, then, puts 
the results of both into the binary function and returns its result: 

return [ = ] (auto param) { 

return binary_func(a(param), b(param)); 

}; 

} 

/v /\ 

5. Let's state that we use theA stdA namespace to spare us some typing in 
the main function: 

using namespace std; 


/v 

6. Now, let's combine our twoA predicate functions in another predicate 

/\ 

function, whichA tells if a given string begins with a and ends with b, as 

"ab" does or "axxxb". As the binary function, we 

/\ 

chooseA std:: logicai and. It is a template class that needs to be 



instantiated, so we use it with curly braces in order to instantiate it. Note 
that we don't provide a template parameterA because for this class, 

/y _ 

itA defaults to void. This specialization of the class deduces all 
parameter types automatically: 


int main() 

{ 

auto a_xxx_b (combine( 
logical_and<>{}, 
begins_with_a, ends_with_b)); 

7. We iterate over the standard input A and print all words back to the 
terminal, which satisfies our predicate: 


c°py_if(istream_iterator<string>{cin}, {}, 

ostream_iterator<string>{cout, ", "}, 
a _xxx_b); 
cout << '\n'; 

} 

8. Compiling and running the program yields the following output. We 

/V /V 

feed the program with fourA words, but only twoA satisfy the predicate 
criteria: 

$ echo "ac cb ab axxxb" | ./combine 
ab, axxxb. 

There’s more... 

The STL already provides a useful bunch of functional objects such 

/V 

asA std :: logicai and, std: : logicai or, as well as many others, so we do 
not need to reimplement them in every project. It's a good idea to have a look 
at the C++ reference and explore what's there 
already: http ://en. cppreference. com/w/cpp / utility/functional 



Calling multiple functions with the same 
input 


There are a lot of tasks, which lead to repetitive code. A lot of repetitive code 
can be eliminated easily using lambda expressions and a lambda expression 

/V 

helper thatA wraps such repetitive tasks is created very quickly. 

In this section, we will play with lambda expressions in order to forward a 
single call with all its parameters to multiple receivers. This is going to 
happen without any data structures in between, so the compiler has a 

/V /V 

simpleA job to generateA a binary without overhead. 

How to do it... 

We are going to write a lambda expression helper, which forwards a single 
call to multiple objects, and another lambda expression helper, which 
forwards a single call to multiple calls of other functions. In our example, we 
are going to use this to print a single message with different printer functions: 

1. Let's include the STL header we need for printing first: 

#include <iostream> 

/v 

2. At first, we implement the muiticaiiA function, which is central to this 

/\ 

recipe. ItA accepts an arbitrary number of functions as parameters and 
returns a lambda expression that accepts one parameter. It forwards this 
parameter to all the functions thatA were provided before. This way, we 

/V 

can defineA auto call_all (multicall (f, g, h)),and 

/V 

then,A caiiaii (12 3) leads to a sequence of calls, f (123); g(i23); 

/\ _ /\ 

h (123) ;. A This function looks really complicatedA because we need a 

/V 

syntax trick in order to expand the parameter pack,A functions, into a 

y\ /v 

series of calls by using anA std:: initiaiizer iistA constructor: 


static auto multicall (auto ...functions) 




return [ = ] (auto x) { 

(void)std::initializer_list<int>{ 

( (void)functions(x) , 0) . . . 

}; 

}; 

} 

. /V /V /V 

3. The next helper accepts a function, A f, and a packA of parameters, A xs. 

/V 

What it does is it callsA f with each of those parameters. This way, 

/\ /V 

aA for_each (f, l, 2, 3) call leads to a series of calls:A f (l); f(2); 
f (3);. This function essentially uses the same syntax trick to expand the 

/V 

parameter pack,A xs, to a series of function calls, as the other function 
before: 


static auto for_each (auto f, auto ...xs) { 

(void)std::initializer_list<int>{ 

((void)f(xs), 0)... 

}; 

} 

—, /V 

4. TheA brace print function accepts two characters and returns a new 

/\ /\ /\ 

function object, which accepts one parameter,A x. It willA print A it, 

/\ 

surrounded by the two characters we just capturedA before: 

static auto brace_print (char a, char b) { 
return [=] (auto x) { 

std::cout << a << x << b << ", 

}; 

} 

5. Now, we can finally put everything to use in the main function. At first, 

/V _ 

we define functionsA f, g, and h. They represent print functions that 
accept values and print them surrounded byA different 

_ /V 

braces/parentheses each. TheA ni function takes any parameter and just 
prints a line break character: 


int main() 

{ 

auto f (brace_print(' (' , ')')); 

auto g (brace_print(' [' , ']')); 

auto h (brace_print('{' , '}')); 

auto nl ([](auto) { std::cout << '\n'; }); 



6. Let's combine all of them using our multi call helper: 

auto call_fgh (multicall(f, g, h, nl)); 


7. For each of the numbers we provide, we want to see them individually 
printed three times surroundedA by different pairs of braces/parentheses. 
This way, we can do a single function callA and end up with fiveA calls 

/V /\ 

to our multifunction, which does A another fourA calls to f, g, h, and nl. 

for_each(call_fgh, 1, 2, 3, 4, 5) ; 

} 


/v 

8. Before compiling and running, t hink about what output to expect:A 


$ ./multicaller 


(1), 

[1] , 

{1} 

(2) , 

[2] , 

{2} 

(3) , 

[3] , 

{3} 

(4) , 

[4] , 

{4} 

(5) , 

[5] , 

{5} 


How it works... 


The helpers we just implemented look horribly complicated. This 

/V 

isA because we expand parameter packs with std:: initiaiizer iist. Why 
did we even use that data structure? Let's have a look at for each again: 

auto for_each ([](auto f, auto ...xs) { 

(void)std::initializer_list<int>{ 

((void)f(xs), 0) . . . 

}; 

}) ; 

The heart of this function is the f (xs) expression, xs is a parameter pack, and 
we need to unpack it in order to get the individual values out of it and feed 

them to individual f calls. Unfortunately, we cannot just write 

/\ 

f (xs) . . .A using the ... notation, which we already know. 

What we can do is constructing a list of values using 



std::initializer list, which has a variadic constructor. An expression 

/V /V 

SUCh asA return std::initializer_list<int>{f(xs) ...};A does the job, 
but it has downsides. Let's have a look at an implementation of for each 
which does just this, so it looks simpler than what we have: 

auto for_each ([](auto f, auto ...xs) { 

return std::initializer_list<int>{f(xs)...}; 

}) ; 

/v 

This is easier to grasp, but itsA downsides are the following: 

1. It constructs an actual initializer list of return values from all the f calls. 
At this point, we do not care about the return values. 

2. It returns that initializer list, although we want a "fire and forget" 
function, which returns nothing. 

3. It's possible that f is a function, which does not even return anything, in 
which case, this would not even compile. 

The much more complicated f or each function fixes all these problems. It 
does the following things to achieve that: 

1. It does not return the initializer list, but it casts the whole expression to 
voidA USing (void)std::initializer_list<int>{...}. 

/v /v 

2. Within the initializer expression, it wrapsA f (xs) ... into anA (f (xs), 
o) ... expression. This leads to the return value being thrown away, 

while o is put into the initializer list. 

/v 

3. The f (xs) A in the (f (xs), 0) ... expression is again cast to void, so 

/V 

the return value is really not processed anywhereA if it has any. 

Putting all this together unluckily leads to an ugly construct, but it does it's 
work right and compiles with a whole variety of function objects, regardless 

/V 

of whetherA they return anything or what they return. 

A nice detail of this technique is that the order in which the function calls are 

/\ 

applied is guaranteed to be in aA strict sequence. 


Note 



Casting anything using the old C-style notation 

/V 

(void) expression is advised againstA because C++ has its own 
cast operators. We should have used reinterpret_cast<void> 
(expression) instead, but this would have decreased the 
readability of the code further. 



Implementing transformif using 
std::accumulate and lambdas 


/V /\ 

Most developersA who have used std: : copy^if and std:: transformA may 
have asked themselves already, why there is no std:: transform if. 

/V /V 

TheA std:: copy ifA function copies items from a source range to a 

/\ /\ 

destination range, but skips A the items thatA are not selected by a user- 
defined predicate function. TheA std:: transform unconditionally copies all 
items from a source range to a destination range but transforms them in 
between. The transformation is provided by a user-defined function, which 

might do simple things, such as multiplying numbers or transforming 

/\ 

items A to completely different types. 

/V 

SuchA functions have been there for a long time now, but there is still no 
std:: transform if function. In this section, we are going to implement this 
function. It would be easy to do this by just implementing a function 

/V /\ 

thatA iterates over the ranges while copying all the items thatA are selected 
by a predicate function and transforming them in between. However, we'll 
use this occasion to delve deeper into lambda expressions. 

How to do it... 

We are going to build our own transform if function which works by 
supplying std: accumulate with the right function objects: 

1. We need to include some headers, as always: 

#include <iostream> 

#include <iterator> 

#include <numeric> 

/v 

2. First, we willA implement a function called map. It accepts an input- 

/V 

transforming function as parameter A and returns a function object, 
which works well together with std: accumulate: 




template <typename T> 
auto map(T fn) 

{ 


/v 

3. What we return is a function object thatA accepts a reduce function. 
When this object is called with such a reduce function, it returns another 

/V 

function object, which accepts an accumulator A and an 

/y /y 

inputA parameter. It calls the reduce function on this accumulatorA and 

/y 

the f nA transformed input variable. Don't worry if this looks 
complicated, we'll put it together later and see how it really works: 

return [=] (auto reduce_fn) { 

return [=] (auto accum, auto input) { 

return reduce_fn(accum, fn (input)); 

}; 

}; 

} 

/V 

4. NowA we implement a function called filter. It works exactly the 
same way as the map function, but it leaves the input untouched , while 

/y 

the map function transforms A it using a transform function. Instead, we 
accept a predicate function and skip input variables without reducing 

/V 

them in caseA they are not accepted by the predicate function: 

template <typename T> 
auto filter (T predicate) 

{ 

5. The two lambda expressions have exactly the same function signature as 
the expressions in the map function. The only difference is that the 

/y _ 

inputA parameter is left untouched. The predicate function is used to 
distinguish if we call the reduce fn function on the input or if we just 
reach the accumulator forward without any change: 

return [=] (auto reduce_fn) { 

return [=] (auto accum, auto input) { 
if (predicate(input)) { 

return reduce_fn(accum, input); 

} else { 

return accum; 



} 


} 


}; 


}; 


6. Now let's finally use those helpers. We instantiate iterators that let us 
read integer values from the standard input: 

int main() 

{ 

std::istream_iterator<int> it {std::cin}; 
std::istream_iterator<int> end_it; 

/v /v 

7. Then we define a predicate function,A even, which just returns trueA if 

we have an even number. The transformation function twice multiplies 

/\ 

its integer parameterA with the factor 2 : 

auto even ([](int i) { return i % 2 == 0; }); 
auto twice ([](int i) { return i * 2; }); 

8. The std:: accumulate function takes a range of values and accumulates 

/\ 

them. AccumulatingA means summing the values up with the + operator 
in the default case. We want to provide our own accumulation function. 
This way, we do not maintain a sum of the values. What we do is we 

/V 

assign each value of the range to the dereferenced iterator, A it, and then 
return this iterator after advancing it further: 

auto copy_and_advance ([](auto it, auto input) { 

*it = input; 
return ++it; 

}) ; 

9. Now we finally put together the pieces. We iterate over the standard 

/\ /\ 

inputA and provide an output,A ostream iterator, which prints to the 
terminal. The copy and advance function object works on that output 
iterator by assigning the user input integersA to it. Assigning to the 
output iterator effectively prints the assigned items. But we only want 
the even numbers from the user input, and we want to multiply them. To 

/y 

achieve this, we wrap the copy and advance functionA into an 



/\ /\ 

evenfilterA and thenA into a tw i c.QTYiajjp . 

std::accumulate(it, end_it, 

std::ostream_iterator<int>{std::cout, ", "}, 
filter(even)( 
map(twice) ( 

c°py_and_advance 

) 


std::cout << '\n'; 

} 


10. Compiling and running the program leads to the following output. The 
values 1,3, and 5 are dropped because they are not even, and the values 
2,4, and 6 are printed after they have been doubled: 

$ echo "12345 6" | ./transform^if 

4, 8, 12, 

How it works... 


/v 

This recipe looks really complicatedA because we are nesting lambda 

/\ 

expressions a lot. In order to understand how thisA works, let's first have a 

/V _ /\ 

look at the inn er workings ofA std:: accumulate. This is how it willA look 
like in a typical STL implementation: 

template <typename T, typename F> 

T accumulate(InputIterator first. InputIterator last, T init, F f 
{ 

for (; first != last; ++first) { 
init = f(init, *first); 

} 

return init; 

} 

The function parameter,A f, does the main work here, while the loop collects 
its results in the user provided init variable. In a usual example case, the 

/V 

iterator range may represent a vector of numbers, such asA o, l, 2, 3, 4, 
and the init value is o. TheA f A function is then just a binary function 



/V 

thatA might calculate the sum of two items using the + operator. 

/V 

In thisA example case, the loop just sums up all the items into the 

/V /V 

init A variable, such asA in init = (((0 + l) + 2 ) +3) + 4. Writing it 
down like this makes obvious that std: accumulate is just a general folding 
function. Folding a range means applying a binary operation to an 

accumulator variable and stepwise every item contained in the range (the 

/\ 

result of each operation is then the accumulator value for the next one).A As 
this function is so general, we can do all kinds of things with it, just like 
implementing std:: transform if ! TheA f A function is then also called the 
reduce function. 

/V 

A very direct implementation of transform if A will look as follows: 


template <typename InputIterator, typename OutputIterator, 
typename P, typename Transform> 

OutputIterator transform^if(InputIterator first. InputIterator la 

OutputIterator out, 

P predicate. Transform trans) 

{ 

for (; first != last; ++first) { 
if (predicate (*first)) { 

*out = trans(*first); 

++out; 

} 

} 

return out; 

} 

This looks quite similar to std:: accumulate, if we regard the parameter out 
as the init variable, and somehow get function f to substitute the if-construct 
and its body! 

We actually did that. We constructed that if-construct and its body with the 
binary function object we provided as a parameter to std:: accumulate: 


auto copy_and_advance ((](auto it, auto input) { 
*it = input; 
return ++it; 

}) ; 


/V /V 

The std:: accumuiateA function puts theA init variable into the binary 



function's it parameter. The second parameter is the current value from the 

/V 

source range per loop iteration step. We provided anA output iterator as the 
init parameter of std: : accumulate .. This way, std: : accumulate does not 
calculate a sum, but forwards the items it iterates over to another range. 

/V /V 

ThisA means that we just reimplemented std:: copyA without any predicate 
and transformation, yet. 

The filtering using a predicate was added by us by wrapping the 
copy and advance function object into another function object, which 
employs a predicate function: 


template ctypename T> 
auto filter(T predicate) 

{ 

return [=] (auto reduce_fn) { 

return [=] (auto accum, auto input) { 
if (predicate(input)) { 

return reduce_fn(accum, input); 

} else { 

return accum; 

} 

}; 

}; 

} 

This construction does not look too simple at firstA but have a look at the 

/V 

if A construct. If the predicate function returns true, it forwards the 

/v 

parameters to the reduce_fnA function, which is copy and advance in our 
case. If the predicate returns false, the accum variable, which is the init 

/v 

variable ofA std:: accumulate, is just returned without change. This 

_ A 

implements the skipping part of a filter operation. The if A construct is 
located within the inner lambda expression, which has the same binary 
function signature as the copy and advance function, which makes it a 
fitting substitute. 

/V 

Now we are able to filter A but are still not transforming. This is done with 
the map function helper: 


template <typename T> 
auto map(T fn) 

{ 



return [=] (auto reduce_fn) { 

return [=] (auto accum, auto input) { 

return reduce_fn(accum, fn(input)); 

}; 

}; 

} 

This code looks much easier. It again contains an inner lambda expression, 
which has the same signature as copy and advance has, so it can substitute 
it. The implementation just forwards the input valuesA but transforms the 

/V /\ 

right parameter of the binary function call with theA f nA function. 

Later, when we used those helpers, we wrote the following expression: 


filter (even) ( 
map(twice) ( 

copy_and_advance 



/v 

The filter (even) call captures the even predicateA and gives us a function, 
which takes a binary function in order to wrap it into another binary function, 

_ /V 

which does additional filtering. The map (twice) functionA does the same 

/\ /\ 

with the twice transformation functionA but A wraps the binary 

/V 

function,A copy and advance, into another binary function, which always 
transforms the right parameter. 

Without any optimization, we would get a horribly complicated nested 

/V 

construction of functions thatA call functions and do only a very little amount 
of work in between. However, it is a very simple task for the compiler to 
optimize all the code. The resulting binary is as simple as if it resulted from a 
more direct implementation of transform if. We pay nothing in terms of 
performance this way. But what we get is a very nice composability of 
functions because we were able to stick the even predicate together with the 

/V 

twice transformation function, nearly as simply as if they were lego A. bricks. 



Generating cartesian product pairs of any 
input at compile time 


Lambda expressions in combination with parameter packs can be used for 
complex tasks. In this section, we will implement a function object 

/V 

thatA accepts an arbitrary number of input parameters and generates the 
cartesian product of this set with itself. 

The cartesian product is a mathematical operation. It is noted as a x b, 

_ A 

meaning the cartesian product of set a and set b. The result is anotherA single 
set, which contains pairs of all item combinations of the sets a and b. The 
operation basically means, combine every item from A with every item from 
B. The following diagramA illustrates the operation: 

B 

12 3 



X 

(x, 1) 

(x, 2) (x, 3) 

A 

y 

(y, l) 

(y, 2) (y, 3) 


z 

(z, 1) 

(z, 2) (z, 3) 


/v 

In A the preceding diagram, if a = (x, y, z ), and b = ( 1 , 2 , 3 ), then the 

/V 

cartesian product is (x, l), (x, 2), (x, 3), (y, l), (y, 2),andsoon.A 






If we decide that a and b are the same set, say ( 1 , 2), then the cartesian 
product of that is (1 , 1 ), (1 , 2), (2, l), and ( 2 , 2 ). In some cases, this 
might be declared redundant, because the combination of items with 
themselves (like in (l, l)) or redundant combinations of (l, 2 ) and ( 2 , 

/V /\ 

l) A may not be needed. In such aA case, the cartesian product can be filtered 
with a simple rule. 

In this section, we will implement the cartesian product without any loops but 
with lambda expressions and parameter pack unpacking. 

How to do it... 

/V /V 

We implement a function object that A accepts a function, A f, and a set of 
parameters. The function object will create the cartesian product of the 

/V /V 

parameter sot, filter out the redundant parts, and call A theA f function with 
each of them: 

1. We only need to include the STL header that is needed for printing: 

#include <iostream> 

2. Then, we define a simple helper function that prints a pair of values, and 
we begin implementing the main function: 

static void print(int x, int y) 

{ 

std::cout << "(" << x << ", " << y << ")\n"; 

} 

int main() 

{ 

3. The hard part starts now. We first implement a helper for the cartesian 

A _ 

function that A we are going to implement in the next step. This function 

/V 

accepts a parameter,A f, which will be the print function when we use 
it later. The other parameters are x and the parameter pack rest. These 
contain the actual items of which we want to have the cartesian product. 

/V 

Look at the f (x, rest) expression: for x=iA and rest=2, 3, 4, this 

/v 

will result in calls such asA f (l, 2); f (l, 3 ); f (l, 4);. The (x < 



rest) test is for removing redundancy in the generated pairs. We will 

/V 

look at this in more detailA later: 


constexpr auto call_cart ( 

[ = ] (auto f, auto x, auto ...rest) constexpr 
(void)std::initializer_list<int>{ 

( ((x < rest) 

? (void)f(x, rest) 

: (void)0) 

, 0 ) . . . 

}; 


}) ; 


{ 


4. The cartesian function is the most complex piece of code in this whole 
recipe. It accepts the parameter pack xs and returns a function object 
that captures it. The returned function object accepts a function 

A /V 

object,A f . For a parameter pack,A xs=i ,2,3, the inner lambda 
expression will generate the following calls: cali cart (f, l, l, 2 , 

3); call_cart(f, 2, 1 , 2 , 3); call_cart(f, 3, 1 , 2 , 3);. From 
that range of calls, we can generate all the cartesian product pairs we 

/V 

need. Note that we use the ... notation for expanding theA xs 

/\ _ 

parameter packA twice, which looks weird at first. The first occurrence 

/\ 

of ... expands the entireA xs parameter pack into the caii cart call. 

The second occurrence leads to multiple caii cart calls with a 

/\ 

differingA second parameter: 


constexpr auto cartesian ([=](auto ...xs) constexpr { 
return [=] (auto f) constexpr { 

(void)std::initializer_list<int>{ 

((void)call_cart(f, xs, xs...), 0)... 

}; 

}; 

}) ; 

5. Now, let's generate the cartesian product of the numeric set l, 2 , 3 and 
print the pairs. Without the redundant pairs, this should result in the 

/V 

number pairs, A (l, 2 ), ( 2 , 3), and ( l , 3). More combinations are not 
possible if we ignore the order and do not want the same number in one 
pair. This means that we do not want (l, l), and consider (l, 2 ) and 



(2 , l ) the same pair. First, we let cartesian generate a function object 
that already contains all possible pairs and accepts our print function. 
Then, we use it to let our print function being called with all these 

/V 

pairs. We declare theA print cart variable, constexpr, so we can 

guarantee that the function object it holds (and all the pairs it generates) 

/\ 

is createdA at compile time: 

constexpr auto print_cart (cartesian(1, 2, 3)); 
print_cart(print) ; 

} 


6. Compiling and running yields the following output, just as expected. 
Play around with the code by removing the (x < xs) conditional in the 
caii cart function and see that we get the full cartesian product with 
redundant pairs and the same number pairs: 

$ ./cartesian_product 

( 1 , 2 ) 

(1, 3) 

(2, 3) 

How it works... 

That was another really complicated-looking lambda expression construct. 
But as soon as we understand this thoroughly, we will not be confused by any 
lambda expression anytime soon! 

/V 

So, let's have a detailed look at it. We shouldA get a mental picture of what 


[ 1 , 2 , 3 ] —► ( 1 , 1 ), ( 1 , 2 ), ( 1 , 3 ) 

[ 1 , 2 , 3 ] —► ( 2 , 1 ), ( 2 , 2 ), ( 2 , 3 ) 

[ 1 , 2 , 3 ] —► ( 3 , 1 ), ( 3 , 2 ), ( 3 , 3 ) 


needs to happen: 


+ i, 

-> 2 , 
+ 3 , 


1 , 2 , 3 



These are three steps: 


1. We take our set 1 , 2 , 3 and compose three new sets from it. The first 

/V /V 

part of each of these sets is consecutivelyA a single itemA from the set, 

/V 

and the second part is the whole set itself.A 

2. We combine the first item with every item from the set and get as many 
pairs out of it. 

/V 

3. From these resulting pairs, we only pick the ones thatA are not 
redundant (as for example ( 1 , 2 ) and ( 2 , 1 ) are redundant) and not 
same-numbered (as for example ( 1 , 1 )). 

Now, back to the implementation: 


constexpr auto cartesian ([=](auto ...xs) constexpr { 
return [=](auto f) constexpr { 

(void)std::initializer_list<int>{ 

((void)call_cart(f, xs, xs...), 0)... 

}; 

}; 

}) ; 

The inner expression,A caiicart (xs, xs...), exactly represents the 

/V 

separation of ( 1 , 2 , 3 ) into those new sets, such asA 1 , [ 1 , 2 , 3 ] . The 

/V /\ 

fullA expression,A ((void) cal i_cart (f, xs, xs...), 0)... with the 
other ... outside, does this separation for every value of the set, so we also 
get 2, [ 1, 2, 3 ] and 3 , [ 1, 2, 3 ] . 

Step 2 and step 3 are done by caii cart: 

auto call_cart ([](auto f, auto x, auto ...rest) constexpr { 
(void)std::initializer_list<int>{ 

(((x < rest) 

? (void)f(x, rest) 

: (void)0) 

, 0) . . . 

}; 

}) ; 

Parameter x always contains the single value picked from the set, and rest 
contains the whole set again. Let's ignore the (x < rest) conditional at first. 
Here, the expression f (x, rest), together with the ... parameter pack 



/\ 

expansion generates the function callsA f ( 1 , 1 ), f ( 1 , 2), and so on, which 
results in the pairs being printed. This was step 2. 

Step 3 is achieved by filtering out only the pairs where (x < rest) applies. 

/\ 

We made allA lambda expressions and the variables holding 

/V 

themA constexpr. By doing so, we can now guarantee that the compiler will 

/V 

evaluate their code at compile time and compile a binary thatA already 
contains all the number pairs instead of calculating them at runtime. Note that 

this only happens if all the function arguments we provide to a constexpr 

/\ 

function areA known at compile time already. 



Chapter 5. STL Algorithm Basics 


We will cover the following recipes in this chapter: 

• Copying items from containers to other containers 

• Sorting containers 

• Removing specific items from containers 

• Transforming the contents of containers 

• Finding items in ordered and unordered vectors 

/\ /\ 

• LimitingA the valuesA of a vector to a specific numeric range with 

std::clamp 

• Locating patterns in strings with std:: search and choosing the optimal 
implementation 

• Sampling large vectors 

• Generating permutations of input sequences 

• Implementing a dictionary merging tool 



Introduction 


The STL does not only contain data structures but also algorithms, A of 

/V /V 

course. While data structures help store A and maintainA data in different 
ways with different motivations and targets, algorithms apply specific 
transformations to the data in such data structures. 

/V 

Let's have a look at a standard task, such asA summing up items from a 
vector. This can be done easily by looping over the vector and summing up 
all the items into an accumulator variable called sum: 

vector<int> v {100, 400, 200 /*, ... */ }; 

int sum {0} ; 

for (int i : v) { sum += i; } 
cout << sum << 1 \n 1 ; 

But because this is quite a standard task, there is also an STL algorithm for 
this: 

cout << accumulate(begin(v), end(v), 0) << 1 \n 1 ; 

In this case, the handcrafted loop variant is not much longer, and it is also not 
significantly harder to read than a one-liner which says what it does: 

/V 

accumulate. In a lot of cases, however, it is awkward to readA a 10-line code 
loop just to realize, "Did I just have to study the whole loop in order to 
understand that it does a standard task, X?", rather than seeing one line of 

code, which uses a standard algorithm whose name clearly states what it 

/\ 

does, SUCh asA accumulate, copy, move, transform, or shuffle. 

/v 

The basic idea is to provide a rich variety of algorithms thatA can be used by 
programmers on a daily basis in order to reduce the need to repeatedly 
reimplement them. This way, programmers can just use off the shelf 

/V 

algorithm implementations and concentrate on the new A problems, instead of 
wasting time on problems thatA already have been solved by the STL. 



Another perspective is correctness—if a programmer implements the same 
thing again and again for a hundred times, there is some probability that this 

/V 

may introduce a slightA error in one or the other attempt. This would be 

/V 

completely unnecessary and also very embarrassingA if, for example, it is 
pointed out by a colleague during code review, whereas at the same time, a 
standard algorithm could have been used. 

Another important point of STL algorithms is efficiency. Many STL 
algorithms provide multipleA specialized implementations of the same 
algorithm, which do things differently, depending on the iterator type they 
are being used with. For example, if all the elements in a vector of integers 
should be zeroed, this can be done with the STL algorithm std:: fill. 
Because the iterator of a vector can already tell the compiler that it iterates 

/V 

over contiguous memory, it can select theA implementation of 

/\ /\ 

std: : f iiiA which uses the C procedureA memset. If the programmer 
changes the container type from vector to list, then the STL algorithm 

/V 

cannot use memset any longer A and has to iterate over the list in order to zero 

/V 

the items individually. In case the programmer uses A memset himself, the 
implementation would be unnecessarily hardcoded to using vectors or arrays 
because most other data structures do not save their data in contiguous 
memory chunks. In most cases, it makes little sense to try to be smart, as the 
implementers of the STL may already have implemented the same ideas, 
which can be used for free. 

Let's summarize the preceding points. Using STL algorithms is good for: 

• Maintainability: The names of the algorithms already state in a 
straightforward manner what they do. Explicit loops are rarely both 

/V 

better to readA and as data-structure agnostic as standard algorithms. 

• Correctness: The STL has been written and reviewed by experts, and 
used and tested by so many people that you are pretty unlikely to reach 
the same degree of correctness when reimplementing the complex parts 
of it. 

• Efficiency: STL algorithms are, by default, at least as efficient as most 
handcrafted loops. 

Most algorithms work on iterators. The concept of how iterators work is 
already explained in Chapter 3 . A Iterators. In this chapter, we'll concentrate 



on using STL algorithms for different problems in order to get a feeling 

/V 

ofA how they can be profitably put to use. Showing all STL algorithms 

would blow up this book to a very boring C++ reference, although there is 

/\ 

already a C++ reference publicly available.A 

The best way to become an STL ninja is having the C++ reference always at 
hand or, at least, saved in a browser bookmark. When solving a task, every 
programmer should have a look at it with the question back in his mind, "Is 

/V 

there an STL algorithm for my problem?", before writing code himself.A 
A very good and complete C++ reference is available for online viewing at: 
http://cppreference.com 
It can also be downloaded for offline viewing. 

Note 

In job interviews, good fluency with the STL algorithms is often 

/\ /V 

regardedA as an indicator of aA strong knowledge of C++. 



Copying items from containers to other 
containers 


The most important STL data structures have iterator support. This means 
that it is at least possible to get iterators via begin () and end () functions, 
which point to the data structure's underlying payload data and allow to 
iterate over that data. The iteration always looks the same, no matter what 
kind of data structure is iterated over. 

We can get iterators from vectors, lists, deques, maps, and so on. Using 
iterator adaptors, we can even get iterators as an interface to files, standard 

/V 

input, and standard output. Moreover,A as we saw in the previous chapter, 
we can even wrap iterator interfaces around algorithms. Now, where we can 
access everything with iterators, we can combine them with STL algorithms, 
which accept iterators as parameters. 

A really nice way to show how iterators help abstract the nature of different 

/V 

data structures awayA is the std: : copy algorithm, which just copies items 
from one set of iterators to an output iterator. Where such algorithms are 
used, the nature of the underlying data structure is not really relevant any 

/V 

longer. In order to demonstrate this, we will play a bit withA std: : copy. 

How to do it... 


In this section, we will use different variants of std: 


: copy. 


1. Let's first include all headers we need for the data structures we use. 

/\ /\ 

Additionally, we declare that we use theA stdA namespace: 


#include <iostream> 
#include <vector> 
#include <map> 
#include <string> 
#include <tuple> 




#include <iterator> 
#include <algorithm> 

using namespace std; 


2. We will use pairs of integer and string values in the following. In order 
to nicely print them, we should first overload the « stream operator for 
them: 

namespace std { 

ostreamS operator«(ostream &os, const pair<int, string> & 
{ 

return os << "(" << p.first << ", " << p.second << ")" 

} 

} 

3. In the main function, we fill a vector of integer-string pairs with some 

/V 

default values. And we declare a map A variable, which associates integer 
values with string values: 

int main() 

{ 

vector<pair<int, string>> v { 

{1, "one"}, {2, "two"}, {3, "three"}, 

{4, "four"}, {5, "five"}}; 

map<int, string> m; 

4. Now, we use std: : copy n to copy exactly three integer-string pairs 
from the front of the vector to the map. Because vectors and maps are 
completely different data structures, we need to transform the items 

/V 

from the vector using theA insert iterator adapter. The 

std:: inserter function produces such an adapter for us. Please be 

always aware that using algorithms like std: : copy n combined with 

/V 

insert iterators isA the most generic way to copy/insert items to other 
data structures, but not the fastest. Using the data structure-specific 
member functions for inserting items is usually the most efficient way: 

c°py_ n (begin(v), 3, inserter(m, begin(m))); 


5. Let's print what's in the map afterward. Throughout the book, we have 
often been printing a container's content using the std:: copy function. 



The std:: ostream iterator helps a lot in that regardA because it 
allows us to treat the user shell's standard output as another container 
we can copy data into: 

auto shell_it (ostream_iterator<pair<int, string>>{cou 


copy(begin(m), end(m), shell_it); 
cout << 1 \n 1 ; 

6. Let's clear the map again for the next experiment. This time, we 

/V /\ 

move A items from the vector to the map, and this time, it's all A the 
items: 


m.clear(); 

move(begin(v), end(v), inserter(m, begin(m))); 

/\ 

7. We print the new content of the map again. Moreover, A as 

/\ /v 

std:: move A is an algorithm thatA also alters the data source, we will 
print the source vector too. This way, we can see what happened to it 
when it acted as a move source: 

copy(begin(m), end(m), shell_it); 
cout << 1 \n 1 ; 

copy(begin(v), end(v), shell_it); 
cout << 1 \n 1 ; 

} 


8. Let's compile and run the program and see what it says. The first two 
lines are simple. They reflect what the map contained after applying the 
copy n and move algorithms. The third line is interestingA because it 

/V 

shows that the strings in the vector thatA we used as move source are 
now empty. This is because the content of the strings has not been 
copied but efficiently moved (which means that the map uses the string 
data in heap memory that was previously referenced by the string 

/V 

objects in the vector^. We should usually not access items thatA were a 

/V 

move sourceA before we reassigned them, but let's ignore that for the 



sake of this experiment: 


$ ./copying_items 


1, one) , 

(2, 

two) , 

(3, 

three), 


1, one) , 

(2, 

two) , 

(3, 

three), (4, four), 

(5, five). 

1, ), (2, 

) , 

(3, ), 

(4, 

), (5, ), 



How it works... 


As std :: copy is one of the simplest STL algorithms, its implementation is 
very short. Let's have a look at how it could be implemented: 

template <typename InputIterator, typename OutputIterator> 
OutputIterator copy(InputIterator it. InputIterator end^it. 

OutputIterator outfit) 

{ 

for (; it != end_it; ++it, ++out_it) { 

*out_it = *it; 

} 

return out_it; 

} 


This looks exactly as one would implement the copying of items from one 
iterable range to the other by hand, naively. At this point, one could also ask, 
"So why not implementing it by hand, the loop is simple enough and I don't 
even need the return value?", which is, of course, a good question. 

While std:: copy is not the best example for making code significantly 
shorter, a lot of other algorithms with more complex implementations are. 
What is not obvious is the hidden automatic optimization of such STL 

/V 

algorithms. If we happen to use std: : copy with data structures thatA store 

/V 

their items in contiguous memory (asA std:: vector and std:: array do), 
and the items themselves are trivially copy assignable , then the compiler will 
select a completely different implementation (which assumes the iterator 
types to be pointers): 


template <typename InputIterator, typename OutputIterator> 
OutputIterator copy(InputIterator it. InputIterator end_it. 

OutputIterator outfit) 


{ 


const size t num items (distance(it, end it)); 



} 


memmove(out_it, it, num^items * sizeof(*it)); 
return it + num items; 


This is a simplified version of how the memmove variant of the std:: copy 
algorithm can look in a typical STL implementation. It is faster than the 

/V 

standard loop version, and this time, it is also not asA nice to read. But 
nevertheless, std: : copy users automatically profit from it if their argument 
types comply with the requirements ofA this optimization. The compiler 
selects the fastest implementation possible for the chosen algorithm, while 
the user code nicely expresses what the algorithm does without tainting the 
code with too many details of the how. 

STL algorithms often simply provide the best trade-off between readability 

/V 

and optimal implementation.A. 

Note 

Types are usually trivially copy assignable if they only consist of 
one or multiple (wrapped by a class/struct) scalar types or classes, 

/V 

which can safely be moved using memcopy/memmoveA without the 
need to invoke a user-defined copy assignment operator. 

We also used std: :move. It works exactly like std:: copy, but it applies 
std::move (*it) to the source iterator in the loop in order to cast lvalues to 
rvalues. This makes the compiler select the move assignment operator of the 
target object instead of the copy assignment operator. For a lot of complex 

/V 

objects, this performs betterA but destroys the source object. 



Sorting containers 


—I 


Sorting values is quite a standard task, and it can be done in various ways. 
Every computer science student who was tortured with having to learn a 
majority of existing sorting algorithms (together with their performance and 
stability trade-offs for exams) knows that. 

Because this is a solved problem, programmers should not waste their time 

/V 

inA solving it again, except if it is for learning purposes. 

How to do it... 


In this section, we are going to play with std:: sort and 
std::partial_sort: 


/\ 

1. First, we include all that's necessaryA and declare that we use the 

/V 

stdA namespace: 

#include <iostream> 

#include <algorithm> 

#include <vector> 

#include <iterator> 

#include <random> 

using namespace std; 


2. We will print the state of a vector of integers multiple times, so let's 
abbreviate this task by writing a small procedure: 


static void print(const vector<int> &v) 

{ 

copy(begin(v), end(v), ostream_iterator<int>{cout, " 
cout << 1 \n 1 ; 

} 





/y 

3. We begin with a vector thatA contains some example numbers: 


int main() 

{ 

vector<int> v {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}; 

4. Because we will shuffle the vector multiple times in order to play with 
different sort functions, we need a random number generator: 

random_device rd; 
mtl9937 g {rd()}; 

5. The std: : issorted function tells us if the content of a container is 
sorted. This line should print l: 

cout << is_sorted(begin(v), end(v)) << '\n'; 

6. With std: : shuffle, we shake around the content of the vector in order 
to sort it again later. The first two arguments denote the range that will 
be shuffled and the third argument is the random number generator: 

shuffle(begin(v), end(v), g); 

7. The is sortedA function should now return false so that o is printed, 
and the values in the vector should be the same but in a different order. 
We will see after we have printed both again to the shell: 

cout << is_sorted(begin(v), end(v)) << '\n'; 
print(v); 

8. Now, we reestablish the original item ordering by using std: : sort. The 
same prints to the terminal should now again give us the sorted ordering 
from the beginning: 

sort (begin(v), end(v)); 

cout << is_sorted(begin(v), end(v)) << '\n'; 
print(v); 

9. Another interesting function is std: partition. Maybe, we do not want 

/V 

to fully sort the listA because it is sufficient to just have the items 



/V 

thatA are smaller than some value at the front. So, let's partition the 

/\ 

vector in order to move all the items that are smaller than 5 A to the front 
and print it: 


shuffle(begin(v), end(v), g) ; 

partition(begin(v), end(v), [] (int i) { return i < 5; 
print(v); 

10. The next sort-related function is std: :partiai_sort. We can use it to 

/v 

sort the content of a container, but only to some extent. It will putA the n 
smallest of all vector elements in the first half of the vector in a sorted 
order. The rest will reside in the second half, which will not be sorted: 

shuffle(begin(v), end(v), g); 

auto middle (next(begin(v), int(v.size()) / 2)); 

partial_sort(begin(v), middle, end(v)); 

print(v); 

/\ 

11. What if we want to sort a data structure thatA has no comparison 
operator? Let's define one and make a vector of such items: 

struct mystruct { 
int a; 
int b; 

} ; 


vector<mystruct> mv {{5, 100}, {1, 50}, {-123, 1000}, 

{3, 70}, {-10, 20}}; 


/v 

12. The std:: sort A function optionally accepts a comparison function as 
its third argument. Let's use that and provide it with such a function. Just 

/V 

to show that this is possible, we compare them by their second field, A b. 

/v 

This way, they will appear in the order of mystruct: :bA and not 

mystruct::a: 


sort(begin(mv) , end(mv), 

[] (const mystruct &lhs, const mystruct &rhs) { 



return lhs.b < rhs.b; 

}) ; 

13. The last step is printing the sorted vector of mystruct items: 

for (const auto &[a, b] : mv) { 

cout « "{" « a « ", " « b « "} "; 

} 

cout << '\n'; 

} 

/v 

14. Let's compile and run our program. The first lA results from the 
std : : is sorted call after initializing the sorted vector. Then, we 
shuffled the vector and got a o from the second is sorted call. The 
third line shows all the vector items after the shuffling. The next 1 is the 
result of the issorted call after sorting it again with std: : sort. Then, 

/V 

we shuffled the whole vector again andA partitioned it using 

std: : partition. We can see that all the items that are less than 5 are 

also to the left of 5 in the vector. All items that are greater than 5 are to 

/\ _ /\ 

itsA right. Apart from that, they seem shuffled. The second lastA line 
shows the result of std: :partiai_sort. All items up to the middle 
appear strictly sorted but the rest do not. In the last line, we can see our 
vector of mystruct instances. They are strictly sorted by their second 
member values: 


$ ./sorting_containers 
1 
0 

7, 1, 4, 6, 8, 9, 5, 2, 3, 10, 

1 

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 

1, 2, 4, 3, 5, 7, 8, 10, 9, 6, 

1, 2, 3, 4, 5, 9, 8, 10, 7, 6, 

{-10, 20} {1, 50} {3, 70} {5, 100} 


{-123, 1000} 


How it works... 

We have used different algorithms, which have to do with sorting: 




Algorithm 


Purpose 


/V 

std:: sort Accepts a range as argumentsA and simply sorts it. 

/V /V 

std -- is sorted Accepts a range as argument A and tells if A that range is 

sorted. 

This is, kind of, the reverse operation to sorting; it 
std:: shuffle accepts a range as arguments and shuffles its items 

around. 

Accepts a range as arguments and another iterator, 

/\ 

^ ^ which tells until where the input range shouldA be 

std::partial_sort F ° 

sorted. Behind that iterator, the rest of the items appear 
unsorted. 

Accepts a range and a predicate function. All items for 

/V 

std::partition which the predicate function returns trueA are moved to 

the front of the range. The rest is moved to the back. 

/V 

For objects that A do not have a comparison operator < implementation, it is 

possible to provide custom comparison functions. These should always have 

/\ 

a signature SUCh as A bool function_name (const T &lhs, const T 
/\ /\ 

&rhs ) A and shouldA not have any side effects during execution. 

There are also other algorithms such asA std:: stabie sort, which also sort 
but preserve the order of items with the same sort key and 

std::stable_partition. 


Note 



std:: sort has different implementations for sorting. Depending 
on the nature of the iterator arguments, it is implemented as 

selection sort, insertion sort, merge sort, or completely optimized 

/\ /\ 

for a smaller A number A of items. On the user side, we usually do 
not even need to care. 



Removing specific items from containers 


/V 

Copying, transforming, and filtering are perhapsA the most 
commonA operations on ranges of data. In this section, we concentrate on 
filtering items. 

Filtering items out of data structures, or simply removing specific ones, 
works completely differently for different data structures. In linked lists (such 
asA std:: list), for example, a node can be removed by making its 
predecessor point to its successor. After a node is removed fromA the link 

/V 

chain in this way, it can be given back to the allocator.A In contiguously 
storing data structures (std:: vector, std::array, and, to some 

/V 

extent,A std:: deque), items can only be removed by overwriting them with 

/v 

other items. If an item slot is marked to be removed, all the items that A are 

/V 

behind it A must be moved one slot further to the front in order to fill the gap. 
This sounds like a lot of hassle, but if we want to simply remove whitespace 
from a string, for example, this should be achievable without much code. 

When having either data structure at hand, we do not really want to care how 
to remove an item. It should just happen. This is what std:: remove and 
std: : remove if Can do for US. 


How to do it... 


We will transform a vector's content by removing items in different ways: 


1. Let's import all the needed headers and declare that we use 

/\ /V 

theA stdA namespace: 


#include 

#include 

#include 

#include 


<iostream> 

<vector> 

<algorithm> 

<iterator> 


using namespace std; 




2. A short print helper function will print our vector: 


void print(const vector<int> &v) 

{ 

copy(begin(v), end(v), ostream_iterator<int>{cout, " 
cout << '\n'; 

} 


3. We'll begin with an example vector containing some simple integer 
values. We'll also print it, so we can see how it changes with the 
function we apply to it later: 


int main() 

{ 

vector<int> v {1, 2, 3, 4, 5, 6}; 
print(v); 

4. Now let's remove all the items with the value 2 from the vector. 

/V /\ 

std:: remove moves the other items in a way that the one valueA 2 A we 
actually have in the vector vanishes. Because the vector's actual content 
is shorter after removing items, std:: remove returns us an iterator 
pointing to the new end. The items between the new end iterator and the 
old end iterator are to be considered garbage, so we tell the vector to 

erase them. We surround the two removal lines with a new scope 

/\ 

because the new end iterator is invalidated afterwardA anyway, so it can 
go out of scope immediately: 

{ 

const auto new_end (remove(begin(v), end(v), 2)); 
v.erase(new_end, end(v)); 

} 

print(v); 

/v 

5. Now let's remove all theA odd numbers. In order to do so, we implement 

/V 

a predicate, which tells us if a number is oddA and feed it into the 
std: : remove if function, which accepts such predicates: 

{ 

auto odd_number ([](int i) { return i % 2 != 0; }) 

const auto new_end ( 

remove_if(begin(v), end(v), odd_number)); 
v.erase(new end, end(v)); 



} 

print(v); 


6. The next algorithm we try out is std:: replace. We use it to overwrite 

/\ _ /\ /V 

all values of 4 with the valueA 123. TheA std: : repiaceA function also 
exists as std: :repiace_if, which also accepts predicate functions: 

replace(begin(v), end(v), 4, 123); 
print (v); 

7. Let's pump completely new values into the vector and create two new 
empty vectors in order to do another experiment with those: 

V = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}; 

vector<int> v2; 
vector<int> v3; 


8. Then, we implement a predicate for odd numbers again and another 
predicate function, which tells the opposite if a number is even: 

auto odd_number ([](int i) { return i % 2 != 0; }); 
auto even_number ([](int i) { return i % 2 == 0; }); 

9. The next two lines do exactly the same thing. They copy even values to 

_ 

the vectors,A v 2 and v3. The first line does this with the 
std:: remove copy if algorithm, which copies everything from a 
source container to another container which does not fulfill the predicate 
constraint. The other line uses std:: copy if, which copies everything 

/V 

thatA does fulfill the predicate constraint: 


remove_copy_if(begin(v), end(v), 

back_inserter(v2), odd_number); 
copy_if(begin(v), end(v), 

back_inserter(v3), even_number); 

/v 

10. Printing both the vectors should now result inA the same output: 

print(v2); 
print(v3); 



11. Let's compile and run the program. The first output line shows the vector 
after its initialization. The second line shows it after removing all the 

values of 2 . The next line shows the result of removing all the odd 

/\ 

numbers. Before the fourth line, we replaced all the values of 4A with 
123 . The last two lines show vectors v 2 and v3: 

$ ./removing_items_from_containers 


1 , 

2, 

3, 

4, 

5, 

1, 

3, 

4, 

5, 

6, 

4, 

6, 




123 

r 

6, 



2, 

4, 

6, 

8, 

10 , 

2, 

4, 

6, 

8, 

10 , 


How it works... 

We have used different algorithms, which have to do with filtering: 

Algorithm Purpose 

/V 

Accepts a rangeA and a value as arguments and removes 
s td:: remove any occurrence of the value. Returns a new end iterator 

/V 

of the modified range. A 

Accepts a range and two values as arguments and 

/\ 

std:: replace replaces all the occurrences of the first value withA the 

second value. 

/V 

Accepts aA range, an output iterator, and a value as 
std:: remove copy arguments and copies all the values that are not equal to 

/V 

the given valueA from the range to the output iterator. 

/V /V 

Works similar toA std::repiaceA but analogous to 
std::repiace_copy std; . rem ove_copy. The source range is not altered. 



/V 

Works like std:: copyA but additionally accepts a 
predicate function as an argument in order to copy only 

S "C G l ! C Op y 1 T. /v 

the values thatA the predicate accepts, which makes it a 
filter function. 

Note 

For every one of the listed algorithms, there also exists an *_if 
version, which accepts a predicate function instead of a value, 
which then decides which values are to be removed or replaced. 



Transforming the contents of containers 


/V _ /V /V 

IfA std:: copy is the simplest STL algorithm forA application onA ranges, 
std:: transform is the second simplest STL algorithm. Just as copy, it copies 
items from one range to another but additionally accepts a transformation 
function. This transformation function can alter the value of the input type 
before it is assigned to an item in the destination range. Furthermore, it can 
even construct a completely different type, which is useful if the source range 
and destination range differ in their payload item types. It is simple to use but 
still very useful, which makes it an ordinary standard component used in 
portable day-to-day programs. 

How to do it... 


In this section, we are going to use std:: transform in order to modifyA the 
items of a vector while copying them: 

/V 

1. As always, we first need to include all the necessary headersA and to 
spare us some typing, we declare that we use the std namespace: 

#include <iostream> 

#include <vector> 

#include <string> 

#include <sstream> 

#include <algorithm> 

#include <iterator> 

using namespace std; 


2. A vector with some simple integers will do the job as an example source 
data structure: 

int main() 

{ 

vector<int> v {1, 2, 3, 4, 5}; 




3. Now, we copy all the items to an ostream iterator adapter in order to 

/\ /\ 

print them. TheA transformA function accepts a function object, which 
accepts items of the container payload type and transforms them during 
each copy operation. In this case, we calculate the square of each 
number item, so the code will print the squares of the items in the vector 
without us having to store them anywhere: 


transform(begin(v), end(v), 

ostream_iterator<int>{cout, ", "}, 
[] (int i) { return i * i; }); 
cout << '\n' ; 


a /v 

4. Let's do another transformation. From theA number A 3, for example, we 

/V /V 

could generate A a nicely readable string such as A 3 A 2 = 9. The 
following int_to_string function object does just that using the 

std: : stringstream object: 

auto int_to_string ([](int i) { 
stringstream ss; 
ss << i << " A 2 = " << i * i; 
return ss.str (); 

}) ; 


5. The function we just implemented returns us string values from integer 
values. We could also say it maps from integers to strings. Using the 
transform function, we can copy all such mappings from the integer 
vector into a string vector: 


vector<string> vs; 

transform(begin(v), end(v), back_inserter(vs), 
int_to_string) ; 


6. After printing those, we're done: 


} 


copy(begin(vs), end(vs), 

ostream_iterator<string>{cout, "\n"}); 


7. Let's compile and run the program: 


$ ./transforming items in containers 



1, 4, 9, 16, 25, 
1 A 2 = 1 

2 A 2 = 4 

3 A 2 = 9 

4 A 2 = 16 
5 A 2 = 25 


How it works... 


/\ 

The std: : transformA function works exactly like std: : copy but while 
copy-assigning the values from the source iterator to the destination iterator, 
it applies the user-provided transformation function to the value before 

/V 

assigningA the result to the destination iterator. 



Finding items in ordered and unordered 
vectors 


Often, we need to tell if some kind of item exists within some range. And if it 

/V 

does, we often also need to modify itA or to access other data associated with 
it. 

There are different strategies for finding items. If the items are present in a 
sorted order, then we can do a binary search, which is faster than linearly 
going through the items one by one. If it is not sorted, we are stuck with 
linear traversal again. 

The typical STL search algorithms can do both for us, so it's good to know 
them and their characteristics. This section is about the simple linear search 

/V /V 

algorithmA std:: find, the binary search versionA std: :equai_range, and 
their variants. 


How to do it... 


In this section, we are going to use linear and binary search algorithms on a 
small example data set: 

1. We first include all the necessary headers and declare that we use the 
std namespace: 

#include <iostream> 

#include <vector> 

#include <list> 

#include <algorithm> 

#include <string> 

using namespace std; 


2. Our data set will consist of city structs, which just save a city's name, 
and its population count: 




struct city { 

string name; 
unsigned population; 

}; 

/V 

3. Search algorithms need to be able to compare one itemA to the other, so 

/V 

we overload the == operator for theA city struct instances: 

bool operator==(const city &a, const city &b) { 

return a.name == b.name && a.population == b.population 

} 

/v 

4. We also want to print theA city instances, so we overload the stream 
operator,A «: 


ostreamS operator«(ostream &os, const city 
return os << "{" << city.name << ", " 

<< city.population << "}"; 


} 


&city) 


{ 


5. Search functions typically return iterators. These iterators point to the 
item if they found it or, otherwise, to the end iterator of the underlying 
container. In the last case, we are not allowed to access such an iterator 
Because we are going to print our search results, we implement a 
function that returns us another function object, which encapsulates the 
end iterator of a data structure. When used for printing, it will compare 
its iterator argument against the end iterator and then print the item or, 
otherwise, just <end>: 


template <typename C> 

static auto opt_print (const C Scontainer) 

{ 

return [end_it (end(container))] (const auto &item) { 
if (item != end_it) { 

cout << *item << '\n'; 

} else { 

cout << "<end>\n"; 

} 

}; 

} 


6. We start with an example vector of some German cities: 



int main() 

{ 

const vector<city> c { 

{"Aachen", 246000}, 

{"Berlin", 3502000}, 

{"Braunschweig", 251000}, 
{"Cologne", 1060000} 

}; 


7. Using this helper, we build a city printer function, which captures the 
end iterator of our city vector c: 

auto print_city (opt_print(c)); 

8. We use std:: find to find the item in the vector, which saves the city 

/V 

item of Cologne. At first, this search looks pointlessA because we get 
exactly the item we searched for. But we did not know its position in the 
vector before, and the find function returns us just that. However, we 

/V 

could, for example, make the operator == of the city struct that A we 
overloaded only compare the city name, then we could search just using 

/V 

the city name, without even knowing itsA population. But that would 
not be a good design. In the next step, we will do it differently: 

{ 

auto found_cologne (find(begin(c), end(c), 
city{"Cologne", 1060000})); 
print_city(found_cologne); 

} 

9. Without knowing the population count of a city, and also without 

/V /V 

tampering with itsA ==A operator, we can search only by comparing its 

^ /V /V 

name with the vector's content. TheA std:: find if A function accepts a 

/V 

predicate function object instead of a specific A value. This way, we can 
search for the Cologne city item when we only know its name: 

{ 

auto found_cologne (find_if(begin(c), end(c), 

[] (const auto &item) { 

return item.name == "Cologne"; 

}) ) ; 

print_city(found_cologne); 


} 



10. In order to make searching a bit prettier and expressive, we can 
implement predicate builders. The popuiation_higher_than function 

/v 

object accepts a population size and returns us a function that A tells if a 
city instance has a larger population than the captured value. Let's use it 
to search for a German city with more than two million inhabitants in 
our small example set. Within the given vector, that city is only Berlin: 

{ 

auto population_more_than ([](unsigned i) { 
return [=] (const city &item) { 
return item.population > i; 

} ; 

}) ; 

auto found_large (find_if(begin(c), end(c), 
population_more_than(2000000))); 

print_city(found_large); 

} 

11. The search functions we just used, traverse our containers linearly. Thus 
they have a runtime complexity of 0(n). The STL also has binary search 
functions, which work within 0(log(n)). Let's generate a new example 
data set, which just consists of some integer values, and build another 
print function for that: 


const vector<int> v {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}; 
auto print_int (opt_print(v)); 

12. The std:: binary_search function returns boolean values and just tells 
us if it found an item, but it does not return the item itself. It is important 
that the container we are searching in is sortedA because otherwise, 
binary search doesn't work correctly: 

bool contains_7 {binary_search(begin(v), end(v), 7)}; 
cout << contains_7 << '\n 1 ; 

13. In order to get the items we are searching for, we need other STL 
functions. One of them is std: :equai_range. It does not return an 

iterator for the item we found, but a pair of iterators. The first iterator 

/\ 

points to the first itemA that is not smaller than the value we've been 

_ A 

looking for. The second iterator points to the first itemA that is larger 



than it. In our range, which goes from 1 to 10 , the first iterator points to 

/\ 

the actual 7, because it is the first item, thatA is not smaller than 7. The 

/\ /V 

second iterator points to the sA because it's the first item thatA is larger 
than 7. If we had multiple values of 7, both the iterators would, in 

/V 

fact,A represent a subrange of items: 

auto [lower_it, upper^it] ( 

equal_range(begin(v), end(v), 7)); 
print_int(lower_it); 
print_int(upper_it); 

14. If we just need one iterator; we can use std:: lower bound or 

—r-, ^ /v 

std::upper bound. TheA lower boundA function only returns an 

/y 

iterator to the first item thatA is not smaller than what we searched. 

/V /V 

TheA upper boundA function returns an iterator to the first item 

/V 

thatA is larger than what we searched for: 


print_int(lower_bound(begin(v), end(v), 7)); 
print_int(upper_bound(begin(v), end(v), 7)); 

} 

15. Let's compile and run the program to see if the output matches our 
assumptions: 

$ ./finding_items 
{Cologne, 1060000} 

{Cologne, 1060000} 

{Berlin, 3502000} 

1 

7 

8 

7 

8 

How it works... 


These are the search algorithms we have used in this recipe: 


Algorithm 


Purpose 



Accepts a search range and a comparison value as 

/V 

s td : : f ind arguments. Returns an iterator that A points to the first 

item equal to the comparison value. Searches linearly. 

/V 

,, ££ Works like std:: findA but uses a predicate function 

std::find if 

instead of a comparison value. 

Accepts a search range and a comparison value as 
std: :binary_search arguments. Performs a binary search and returns 

trueA if the range contains that value. 

Accepts a search range and a comparison value, and 

/V 

n n , then performs a binary search for the first item thatA is 

std:: lower bound ,, , , 

not smaller than the comparison value. Returns an 
iterator pointing to that item. 

/\ 

Works like std:: lower boundA but returns an iterator 

/\ 

std:: upper bound to the first item thatA is larger than the comparison 

/\ 

value. A 

Accepts a search range and a comparison value and, 
n , then, returns a pair of iterators. The first iterator is the 

std::equal_range r * 

result of std: : lower boundA and the second iterator IS 

the result of std: :upper_bound. 

All these functions accept custom comparison functions as an optional 
additional argument. This way, the search can be customized, as we did in the 
recipe. 

/V 

Let's have a closer look at how std: : equai range works.A Imagine that we 

/v 

have a vector, A v = {o, 1 , 2 , 3 , 4, 5 , 6, 7, 7 , 7 , 8}, and call 
equai range (begin (v) , end ( v) , 7); in order to perform a binary search 



for the value 7. As equai range returns us a pair of lower bound and upper 

/v 

bound iterators, these should afterwardA denote the range {7, 7, 7}, as 

there are so many values of 7 in the sorted vector. Check out the following 

/\ 

diagramA for more clarity: 


01234567778 



begin 




At first, equai range uses the typical binary search approach until it trips 
into the range of values not smaller than the search value. Then, it splits up to 
a lower bound call and an upper bound call in order to bundle their return 

/V 

values in a pair as the A return value. 

In order to get a binary search function, which just returns the first item 

/V 

thatA fits the requirements, we could implement the following: 

template <typename Iterator, typename T> 

Iterator standard_binary_search(Iterator it. Iterator end_it, T v 
{ 

const auto potential_match (lower_bound(it, end_it, value)); 
if (potential_match != end_it && value == *potential_match) { 
return potential_match; 

} 

return end_it; 

} 

This function uses std:: lower bound in order to find the first item not 
smaller than value. The resulting potentiai match can then have three 
different cases it points to: 

• No item is not smaller than value. In this case, it is identical to end it. 

• The first item that is not smaller than vaiueA is also larger than value. 





Therefore we must signal that we did not find it by returning end it. 

• The item that A potentiaimatch points to is equal to value. So, it is 
not only a potential match, but it is an actual match. Therefore we can 
return it. 


/V /V /V 

If our type t does not support theA ==A operator, it must at leastA support 

/V 

theA < operator for the binary search. Then, we can rewrite the comparison to 

! (value < *potential_match) && ! (*potential_match < value) . If it is 

neither smaller, nor larger, then it must be equal. 

One potential reason why the STL does not provide such a function out of the 

/V 

box is the missing knowledge about the possibility that there areA multiple 

/\ 

hits, asA in the diagram where we have multiple values of 7. 

Note 

/V /V 

Note that data structures such asA std: :map, A std:: set, and so 
on have their own find functions. These are, of course, faster than 
the more general algorithms because they are tightly coupled with 
the data structure's implementation and data representation. 



LimitingA the valuesA of a vector to a 
specific numeric range with std:: clamp 


In a lot of applications, we get numeric data from somewhere. Before we can 
plot or otherwise process it, it may need to be normalized because the values 

/V 

differA randomly far from each other. 

Usually, this would mean a little std:: transform call over the data structure 

/V /\ /\ 

thatA holds all these values, combined withA a simple scalingA function. But 
if we do not know how large or small the values are, we need to go through 
the data first in order to find the right dimensions for the scaling function. 

The STL contains useful functions for this purpose: std: :minmax_element 
and std:: clamp. Using these and combining them with some lambda 
expression glue, we can perform such a task easily. 

How to do it... 


In this section, we will normalize the values of a vector from an example 
numeric range to a normalized one in two different ways, one of them using 

/V 

std: :minmax_elementA and One using std: : clamp! 


/v 

1. As always, we first need to include the following headersA and declare 
that we use the std namespace: 

#include <iostream> 

#include <vector> 

#include <algorithm> 

#include <iterator> 

using namespace std; 


/v 

2. We implement a function for later use, whichA accepts the minimum 
and maximum values of a range, and a new maximum so that it can 

/V 

project values from the old range to a smaller A range that we want to 




_ /V 

have. The function object takes such valuesA and returns another 
function object, which does exactly that transformation. For the sake of 
simplicity, the new minimum is o, so no matter what offset the old data 
had, its normalized values will always be relative to o. For the sake of 
readability, we ignore the possibility that max and min could be of the 
same value, which would lead to a division by zero: 


static auto norm (int min, int max, int new_max) 

{ 

const double diff (max - min); 
return [=] (int val) { 

return int((val - min) / diff * new_max); 

} ; 

} 

3. Another function object builder called ciampvai returns a function 

/\ /V 

object thatA captures theA min and max values and calls std:: clamp on 
values with those values, in order to limit their values to this range: 


static auto ciampvai (int min, int max) 

{ 

return [=] (int val) -> int { 

return clamp(val, min, max); 

} ; 

} 

4. The data we are going to normalize is a vector of varying values. This 
could be, for example, some kind of heat data, landscape height, or stock 
prices over time: 


int main() 

{ 

vector<int> v {0, 1000, 5, 250, 300, 800, 900, 321}; 

5. In order to be able to normalize the data, we need the highest and lowest 

_ _/V /V 

values. TheA std: :minmax_eiementA function is of a great help here. It 
returns us a pair of iterators to exactly those two values: 

const auto [min_it, max_it] ( 

minmax_element(begin(v), end(v))); 


6. We will copy all the values from the first vector to a second one. Let's 



instantiate the second vector and prepare it to accept as many new items 
as we have in the first vector: 


vector<int> v_norm; 
v_norm.reserve(v.size() ) ; 

7. Using std:: transform, we copy the values from the first vector to the 
second. While copying the items, they will be transformed with our 
normalization helper. The minimum and maximum values of the old 
vector are o and 1000 . The minimum and maximum values after 
normalization are o and 255 : 

transform(begin(v), end(v), back_inserter(v_norm), 
norm(*min_it, *max_it, 255)); 

8. Before we implement the other normalization strategy, we print what we 
have by now: 


copy(begin(v_norm), end(v_norm), 

ostream_iterator<int>{cout, ", "}); 
cout << '\n'; 

/v 

9. We reuse the same normalized vector with the other helperA ciampvai, 

/V /V 

whichA clamps A the old range to the range with the minimum of 

/\ /\ 

oA and theA maximum of 2 55: 


transform(begin(v), end(v), begin(v_norm), 
ciampvai(0, 255)); 

10. After printing these values too, we're done: 

copy(begin(v_norm), end(v_norm), 

ostream_iterator<int>{cout, ", "}); 
cout << '\n'; 

} 

11. Let's compile and run the program. Having the values reduced to values 

/V 

from o to 2 55, we couldA use them as brightness values for RGB color 
codes, for example: 

$ ./reducing_range_in_vector 
0, 255, 1, 63, 7 6, _ 2 04, 229, 81, 



0, 255, 5, 250, 255, 255, 255, 255, 


12. When we plot the data, we get the following graphs. As we can see, the 
approach where we divide the values by the difference between the min 
and max values is a linear transformation of the original data. The 
clamped graph loses some information. Both variations can be useful in 
different situations: 



x axis 


How it works... 

/V 

Apart from std:: transformA we used two algorithms: 

std: :minmax_eiement simply accepts the begin and end iterators of an input 
range. It loops through the range and records the largest and the smallest 
element on the way to its end. These values are returned in a pair, which we 
then used for our scaling function. 































/V /v 

The std :: ciampA function, in contrast, does not operateA on an iterable 
range. It accepts three values: an input value, a min value, and a max value. 
The output of this function is the input value cut-off in a way that it lies 
between the allowed minimum and maximum. We could also write 

max(min_val, min(max_val, x) ) instead of std :: clamp(x, min_val, 
max val). 



Locating patterns in strings with std::search 
and choosing the optimal implementation 


Searching for a string in a string is a slightly different problem than finding 
one object in a range. On the one hand, a string is, of course, an iterable range 
(of characters) too. On the other hand, finding a string in a string means 
finding a range in another range. And this comes along with multiple 
comparisons per potential match position, so we need some other algorithm 
for that. 


/\ 

std:: string already contains a f indA function, which can do exactly what 

/\ 

we are talking about;A nevertheless we'll concentrate on std:: search in this 
section. Although std:: search might be used on strings mostly, it works on 
all kinds of containers. The more interesting feature of std:: search is that 
since C++17, it has a slightly different additional interface and allows for 
simply exchanging the search algorithm itself. These algorithms are 
optimized and can be freely chosen by the user, depending on what is better 
in which use case. Additionally, we could implement our own search 

/V 

algorithms and plug them into std:: searchA if we ever come up with 
anything better than what is already provided. 

How to do it... 


We will use the new std:: search function with strings and try its different 
variations with searcher objects: 

1. First, we will include all the necessary headers and declare that we use 

/V /\ 

theA stdA namespace: 

#include <iostream> 

#include <string> 

#include <algorithm> 

#include <iterator> 

#include <functional> 




using namespace std; 


2. We will print substrings from the positions the search algorithm returns 
to us, so let's implement a little helper for that: 


template <typename Itr> 

static void print(Itr it, size_t chars) 

{ 

copy_n(it, chars, ostream_iterator<char>{cout}) ; 
cout << '\n' ; 

} 


/v 

3. AA lorem-ipsum style string will work as our example string, within 
which we will search a substring. In this case, this is "eiitr": 


int main () 

{ 

const string long_string { 

"Lorem ipsum dolor sit amet, consetetur" 

" sadipscing eiitr, sed diam nonumy eirmod"}; 
const string needle {"eiitr"}; 

/v /v 

4. The old A std:: searchA interface accepts the begin/end iterators of the 
string within which we are searching a specific substring and the 
begin/end iterators of the substring. It then returns an iterator pointing to 
the substring it was able to find. If it didn't find the string, the returned 
iterator will be the end iterator: 

{ 

auto match (search(begin(long_string), end(long_st 

begin(needle), end(needle))); 

print(match, 5); 

} 

- /v 

5. The C++17 version of std: : search does not acceptA two pairs of 

iterators but one pair of begin/end iterators and a searcher object. The 

/V /V 

A std: : default sear cher A takes the begin/end pair of iterators of the 

/v 

substring that A we are searching for in the larger string: 

{ 

auto match (search(begin(long_string), end(long_st 

default_searcher(begin(needle), end(needle)))); 



print(match, 5); 

} 

6 . The point of this change is that it is easy to switch the search algorithm 

/V /V 

this way. TheA std: :boyer_moore_searcherA usesA the Boyer-Moore 
search algorithm for a faster search: 

{ 

auto match (search(begin(long_string), end(long_st 

boyer_moore_searcher (begin(needle), 

end(needle)))); 

print(match, 5); 

} 

7. The C++17 STL comes with three different searcher object 
implementations. The third one is the B oyer-Moore-Horspool search 
algorithm implementation: 

{ 

auto match (search(begin(long_string), end(long_st 

boyer_moore_horspool_searcher (begin(needle) , 

end(needle)))); 

print(match, 5); 

} 

} 

8 . Let's compile and run our program. We should see the same string 
everywhere if it runs correctly: 

$ ./pattern_search_string 

elitr 

elitr 

elitr 

elitr 


How it works... 


We utilized four different ways to use std:: search in order to get exactly the 
same result. Which one should we prefer in what situation? 

Let's assume our large string within which we search the pattern is called s, 
and the pattern is called p. Then, std: : search (begin (s) , end (s) , 



begin(p), end(p) ); and std::search(begin(s), end(s), 

/\ 

default_searcher (begin (p) , end (p) ) ; A do exactly the same thing. 

The other searcher function objects are implemented with more sophisticated 
search algorithms: 


• std: : def ault_searcher: This redirects to legacy std: : search 

implementation 

• std::boyer moore searcher: This uses theA Boyev-Moove search 
algorithm 

• std: :boyer_moore_horspool_searcher: This analogously uses 
theA Boyer-Moore-Horspool algorithm 

What makes the other algorithms so special? The Boyer-Moore algorithm 
was developed with a specific idea-the search pattern is compared with the 
string, beginning at the pattern's end, from right to left. If the character in the 
search string differs from the character in the pattern at the overlay position 
and does not even occur in the pattern, then it is clear that the pattern can be 

shifted over the search string by its full length. Have a look at the 

/\ 

followingA diagram, where this happens in step 1. If the character 

/V 

beingA currently compared differs from the pattern's character at this position 
but is contained by the pattern, then the algorithm knows by how many 
characters the pattern needs to be shifted to the right in order to correctly 
align to at least that character, and then, it starts over with the right-to-left 
comparison. In the diagram, this happens in step 2. This way, the Boyer- 

Moore algorithm can omit a whole lot of unnecessary comparisons, 

/\ 

compared withA a naive search implementation: 



string: Winter is coming. 

| A M 

Pattern: coming 

0 "r" nonexistent in pattern. 

Shift by whole pattern width. 



coming 

Position of next "o" known. 
Shift by that amount. 



coming 

Match. Compare all items 
right to left. Bingo! 


/V 

Of course, this would have become the new defaultA search algorithm if it 
hadn't brought its own trade-offs. It is faster than the default algorithm, but it 
needs fast lookup data structures in order to determine which characters are 
contained in the search pattern and at which offset they are located. The 
compiler will select differently complex implementations of those, depending 
on the underlying types of which the pattern consists (varying between hash 
maps for complex types and primitive lookup tables for types such 

/V 

asA char). In the end, this means that the default search implementation will 
be faster if the search string is not too large. If the search itself takes some 
significant time, then the Boyer-Moore algorithm can lead to performance 

/V 

gains in the dimension of a constant factor.A 

The Boyer-Moore-Horspool algorithm is a simplification of the Boyer- 
Moore algorithm. It drops the bad character rule, which leads to shifts of the 

/V /V 

whole pattern widthA if a search string character thatA does not occur in the 
pattern string is found. The trade-off of this decision is that it is slightly 
slower than the unmodified version of Boyer-Moore, but it also needs 
fewerA data structures for its operation. 


Note 



Do not try to reason about which algorithm should be faster in a 

specific case. Always measure the performance of your code with 

/\ 

data samples that are typical for your users and baseA your 
decision on the results. 



Sampling large vectors 


/v 

When there areA very large amounts of numeric data that need to be 
processed in some situations, it may not be possible to process it all in 

/V 

feasible time. In such situations, the data could be sampledA in order to 
reduce the total amount of data for further processing, which then speeds up 
the whole program. In other situations, this might be done not to reduce the 
amount of work for processing but for saving or transferring the data. 

A naive idea of samplingA could be to only pick every N th data point. This 

might be fine in a lot of cases, butA in signal processing, for example, it 

/\ 

could lead to a mathematical phenomenon calledA aliasing. If the distance 
between every sample is varied by a small random offset, aliasing can be 
reduced. Have a look at the following diagram, which shows an extreme 

/\ /V 

case A just to illustrate the point-while theA original signal consists of a sine 

/V 

wave, the triangle points on the graph are sampling points thatA are sampled 
at exactly every 100th data point. Unfortunately, the signal has the samey- 
value at these points! The graph which results from connecting the dots looks 
like a perfectly straightA horizontalA line. The square points, however, show 
what we get when we sample every 1 oo + random (-15, + 1 5) points. Here, 

/V /\ /\ 

theA signal still looks very different fromA the original signal, but itA is at 
least not completely gone as in the fixed step size sampling case. 

/V /V 

TheA std:: sampieA function does not add random alterations to sample 
points with fixed offset but chooses completely random points; therefore, it 

/V 

works a bit differently fromA this example: 





x axis 


How to do it... 


We will sample a very large vector of random data. This random data shows 

a normal distribution. After sampling it, the resulting points should 

/\ 

stillA show a normal distribution, which we will check: 

/\ 

1. First, we need to include everything we use A and declare that we use 

/\ /\ 

theA stdA namespace in order to spare us some typing: 

#include <iostream> 

#include <vector> 

#include <random> 

#include <algorithm> 

#include <iterator> 

#include <map> 

#include <iomanip> 

using namespace std; 


/v 

2. It is easier to play around with the codeA if we configure specific 
characteristics of our algorithm in their own constant variables. These 
are the size of the large random vector and the number of samples 

/V 

thatA we are going to take from it: 


int main() 

{ 

const size_t data_points {100000}; 
const size t sample points {100}; 



































/v 

3. The large, randomly filled vector shouldA get numbers from a random 
number generator, which gives out numbers from a normal distribution. 
Any normal distribution can be characterized by the mean value and the 
standard deviation from the mean value: 


const int mean {10}; 
const size_t dev {3}; 

4. Now, we set up the random generator. First, we instantiate a random 
device and call it once to get a seed for the constructor of a random 
generator. Then, we instantiate a distribution object that A applies 
normal distribution to the random output: 


random_device rd; 
mtl9937 gen {rd()}; 

normal_distribution<> d {mean, dev}; 

5. Now, we instantiate a vector of integers and fill it with a lot of random 
numbers. This is achieved using the std: :generate_n algorithm, which 
will call a generator function object to feed its return value into our 

vector using aback inserter iterator. The generator function object 

/\ /\ 

just wraps around theA d (gen) A expression, which gets a random 

/\ 

number from the random deviceA and feeds it into the distribution 
object: 


vector<int> v; 
v.reserve(data_points) ; 

generate_n(back_inserter(v), data_points, 

[&] { return d(gen); }); 

/v 

6 . Now, we instantiate another vector thatA will contain the much smaller 
set of samples: 


vector<int> samples; 
v.reserve(sample_points); 

/\ 

7. The std:: sample algorithm works similar toA std:: copy, but it takes 
two additional parameters: the number of samples , which it shall take 

from the input range, and a random number generator object, which it 

/\ 

will consult to get A random sampling positions: 



sample(begin(v), end(v), back_inserter(samples), 

sample_points, mt19937{random_device{}()}); 


8 . We're already done with the sampling. The rest of the code is for 

displaying purposes. The input data has a normal distribution, and if the 
sampling algorithm works well, then the sampled vector should show a 

normal distribution too. To see how much of a normal distribution is 

/\ 

left, we willA print a histogram of the values: 

map<int, size_t> hist; 

for (int i : samples) { ++hist[i]; } 


/v 

9. Finally, we loop over all the items in order to print ourA histogram: 


} 


for (const auto &[value, count] : hist) { 
cout << setw(2) << value << " " 

<< string(count, 1 *') << '\n 1 ; 

} 


10. After compiling and running the program, we see that the sampled 

/V 

vector still roughly showsA the characteristics of a normal distribution: 



$ ,/sampling_vectors 
1 * 

3 * 

4 ** 

5 ***** 

0 ***** 

7 ******* 

0 ************** 

H ********* 

12 ******** 

13 ******* 

14 *** 

15 *** 

10 *** 


How it works... 


The std:: sampieA algorithm is a new algorithm, which came with C++17. 
Its signature looks like this: 


template<class Initerator, class Outlterator, 
class Distance, class UnitormRandomBitGenerator> 

Outlterator sample(Initerator first. Initerator last, 

Samplelterator out. Distance n, 

UniformRandomBitGenerator&& g) ; 

/v /\ 

The input range is denoted by the A first and last A iterators, while out is 
the output operator. These iterators have exactly the same function as in 

std:: copy; items are copied from one range to the other. 

/\ /\ 

TheA std:: sampieA algorithm is special in the regard that it will copy only a 

/y /v 

part of the input range A because it samples only nA items. It uses uniform 
distribution internally, so every data point in the source range gets chosen 
with the same probability. 




Generating permutations of input sequences 


/V 

When testing code thatA must deal with sequences of inputs where the order 

/V 

of the arguments is not important, it is beneficial to test whetherA it results in 

the same output for all possible permutations of that input. Such a test could, 

/\ /\ 

for example, check whetherA a self-implementedA sort algorithm sorts 
correctly. 

No matter for what reason we need all permutations of some value range, 
std:: next permutation can conveniently do it for us. We can invoke it on a 
modifiable range, and it changes the order of its items to the next 
lexicographical permutation. 

How to do it... 


/\ 

In this section, we will write a program thatA reads multiple word 

/V 

stringsA from a standard input, and then we will 

/\ 

useA std: :next_permutation to generate and print all the permutations of 
those strings: 

1. First things first again; we include all the necessary headers and declare 

/V /V 

that we use theA stdA namespace: 

#include <iostream> 

#include <vector> 

#include <string> 

#include <iterator> 

#include <algorithm> 

using namespace std; 


2. We begin with a vector of strings, which we feed with the whole 
standard input. The next step is sorting the vector: 




int main() 

{ 

vector<string> v {istream_iterator<string>{cin}, {}}; 

sort (begin(v), end(v)); 


3. Now, we print the vector's content on the user terminal. Afterward, we 
call std : :next_permutation. It systematically shuffles the vector to 

generate a permutation of its items, which we then print again. 

/v 

TheA next_permutation will return false as soon as the last 
permutation was reached: 

do { 

copy(begin(v), end(v), 

ostream_iterator<string>{cout, ", "}); 
cout << 1 2 \n 1 ; 

} while (next_permutation(begin(v), end(v))); 

} 

4. Let's compile and run the function with some example input: 


$ echo "a b c" | ./input_permutations 


a. 

b. 

c 

a. 

c. 

b 

b. 

a. 

c 

b. 

c. 

a 

c. 

a. 

b 

c. 

b. 

a 


How it works... 


The std:: next permutation algorithm is a bit weird to use. This 

/V 

isA because it accepts only a begin/end pair of iterators and then returns true 

/\ 

if it is able to find theA next permutation. Otherwise, it returns false. 

/V /V 

ButA what does theA next permutation even mean? 


The algorithm with which std: :next_permutation finds the next 
lexicographical order of the items, works as follows: 

1. Find the largest index i such that v [i - l] < v [ i ]. If there is none, 
then return false. 

2. Now, find the largest index j such that j >= i and v [ j ] > v [ i - l ] . 



3. Swap the items at position j and position i - 1 . 

4. Reverse the order of the items from position i to the end of the range. 

5. Return true. 

, /v 

The individually permutedA orders we get out of this will always appear in 
the same sequence. In order to see all the possible permutations, we sorted 
the array first, because if we entered "c b a", for example, the algorithm 
would terminate immediately , as this already is the last lexicographic order of 
the elements. 



Implementing a dictionary merging tool 


/v 

Imagine that weA have a sorted list of things, and someone else comes up 
with another sorted list of things, and we want to share the lists with each 
other. The best idea is to combine both the lists. The combination of both the 
lists should be sorted too, as this way, it is easy to look it up for specific 
items. 

Such an operation is also called a merge. In order to merge two sorted ranges 
of items, we would intuitively create a new range and feed it with items from 
both the lists. For every item transfer, we would have to compare the 
frontmost items of our input ranges in order to always select the smallest one 
from what is left from the input. Otherwise, the output range would not be 
sorted any longer. The following diagram illustrates it better: 



/v 

TheA std:: merge algorithm can do exactly that for us, so we do not need to 

/V 

fiddle around too much. In this section, we willA see how to use the 
algorithm. 

How to do it... 

We are going to build up a cheap dictionary of one-to-one mappings from 
















English words to their German translations, and store them in std: : deque 
structures. The program will read such a dictionary from a file and one from 
standard input, and print one large merged dictionary on the standard output 
again. 

1. There are a lot of headers to include this time, and we declare that we 

/V /V 

use theA stdA namespace: 

#include <iostream> 

#include <algorithm> 

#include <iterator> 

#include <deque> 

#include <tuple> 

#include <string> 

#include <fstream> 

using namespace std; 


2. A dictionary entry should consist of a symmetric mapping from a string 
in one language to a string in another language: 

using dict_entry = pair<string, string>; 

3. We are going to both print such pairs to the terminal and read them from 

/\ /\ 

user input, so we need to overload theA « and »A operators: 

namespace std { 

ostreamS operator«(ostream &os, const dict_entry p) 

{ 

return os << p.first << " " << p.second; 

} 

istream& operator»(istream &is, dict_entry &p) 

{ 

return is >> p.first >> p.second; 

} 

} 

4. A helper function that accepts any input stream object will help us in 

/V 

building a dictionary from it. It constructsA std:: deque of dictionary 
entry pairs, and they are all read from the input stream until it is empty. 
Before returning it, we sort it: 



template <typename IS> 

deque<dict_entry> from_instream(IS &&is) 

{ 

deque<dict_entry> d {istream_iterator<dict_entry>{is}, 
sort(begin(d), end(d)); 
return d; 

} 

5. We create two individual dictionary data structures from different input 

/V 

streams. One input stream is opened from theA diet. txt file, which we 
assume to exist. It contains word pairs, line by line. The other stream is 
the standard input: 


int main() 

{ 

const auto dictl (from^instream(ifstream{"diet.txt"})); 
const auto dict2 (from^instream(cin)); 

/v 

6. As the helper function,A f rom instream, has already sorted both the 
dictionaries for us, we can feed them directly into the std:: merge 
algorithm. It accepts two input ranges via its begin/end iterator pairs, 
and one output. The output will be the user shell: 


merge(begin(diet1), end(dictl), 
begin(dict2), end(dict2), 

ostream_iterator<dict_entry>{cout, "\n"}); 

} 

7. We can compile the program now, but before running it, we should 

/V 

create theA diet. txt file with some example content. Let's fill it with 
some English words and their translations to German: 


car auto 

cellphone handy 
house haus 


8. Now, we can launch the program while piping some English-German 
translations into its standard input. The output is a merged and still 
sorted dictionary, which contains the translations of both the inputs. We 
could create a new dictionary file from that: 


$ echo "table tisch fish fisch dog hund" | ./dictionary mer 



car auto 
cellphone handy 
dog hund 
fish fisch 
house haus 
table tisch 


How it works... 

/V 

The std:: mergeA algorithm accepts two pairs of begin/end iterators, which 

denote the input ranges. These ranges must be sorted. The fifth parameter is 

/\ 

an output iterator thatA accepts the incoming items during the merge. 

There is also a variant called std:: inplace merge. This algorithm does the 
same as the other, but it does not need an output iterator because it works in 

/V 

place, as the name already suggests. It takes three parameters: aA begin 
iterator, a middle iterator, and an end iterator. These iterators must all 
reference data in the same data structure. The middle iterator is at the same 
time the end iterator of the first range, and the begin iterator of the second 
range. This means that this algorithm handles a single range, which actually 

/V /V 

consists of two consecutive ranges, such as, A for example, A { a, c, b, d} . 
The first subrange is { a, c } A and the second subrange is { b, d }. 

TheA std:: inpiace mergeA algorithm can then merge both within the same 

/v 

data structure, which results in {A, b, c, d}.A 



Chapter 6. Advanced Use of STL 
Algorithms 

We will cover the following recipes in this chapter: 

• Implementing a trie class using STL algorithms 

/V 

• Implementing a search inputA suggestion generator with tries 

• Implementing the Fourier transform formula with STL numeric 
algorithms 

• Calculating the error sum of two vectors 

• Implementing an ASCII Mandelbrot Tenderer 

• Building our own algorithm - split 

• Composing useful algorithms from standard algorithms - gather 

• Removing consecutive whitespace between words 

• Compressing and decompressing strings 



Introduction 


/v 

In the last chapter, we visited basic STL algorithms and performedA simple 
tasks with them in order to get a feeling ofA the typical STL interface: most 
STL algorithms accept one or more ranges in the form of iterator pairs as 
input/output parameters. They often also accept predicate functions, custom 
comparison functions, or transformation functions. In the end, they mostly 
return iterators again because these can often be fed into some other 
algorithm afterward. 

^ /V /V 

While STL algorithms aimA to be minimal, their interfaces also try to beA as 
general as possible. This enables maximum code reuse potentialA but does 
not always look too pretty. An experienced C++ coder who knows all 
algorithms has a better time reading other people's code if it tries to express 
as many ideas using STL algorithms as possible. This leads to a maximized 
common ground of comprehension between coder and reader. A 
programmer's brain can simply parse the name of a well-known algorithm 
more quickly than it can understand a complex loop, which does a mainly 
similar, but in some detail a slightly different, job. 

At this point, we are using STL data structures so intuitively that we can 
nicely avoid pointers, raw arrays, and other crude legacy structures. The next 
step is lifting our comprehension of STL algorithms up to the levels where 
we can avoid the use of handcrafted loop-control-structure complexes by 
expressing them in terms of well-known STL algorithms. Often, this is a real 
improvement because code becomes simply shorter and more readable while 

/V 

at the same time being A more general and data-structure agnostic. It is 
practically always possible to avoid writing handcrafted loops and taking an 
algorithm out of the std namespace instead, but sometimes, it admittedly 
leads to awkward code. We are not going to differentiate between what is 

/V /V 

awkward and what is not;A we'll onlyA explore the possibilities. 

In this chapter, we will use STL algorithms in creative ways in order to look 
for new horizons and to see how things can be implemented with modern 




/V 

C++. On the way, we willA implement our own STL-like algorithms, which 
can easily be combined with existing data structures and other algorithms 

/V 

designed in the same way. We will also combine A existing STL algorithms to 

get new algorithms, which were not there before. Such combined algorithms 

/\ 

allow for more complex algorithmsA on top of the existing ones, while they 
are themselves extremely short and readable this way. While onA this little 

trip, we will also see where exactly STL algorithms suffer from reusability or 

/\ 

prettiness. Only when we know all A the ways well can we best decide which 
way is the right one. 



Implementing a trie class using STL 
algorithms 


The so-called trie data structure poses an interesting way to storeA data in an 
easily searchable manner. When segmenting sentences of text into lists of 

/V 

words, it is often possible to combine the first few words thatA some 
sentences have in common. 

Let's have a look at the following diagram, where the sentences "hi how are 
you" and "hi how do you do" are saved in a tree-like data structure. The 
first words they have in common are "hi how", and then they differ and split 
up like a tree: 



Because the trie data structure combines common prefixes, it is also called 
prefix tree. It is very easy to implement such a data structure with what the 
STL gives us already. This section concentrates on implementing our own 
trie class. 


How to do it... 

In this section, we will implement our own prefix tree only made from STL 
data structures and algorithms. 

1. We will include all the headers from the STL parts we use and declare 

/V 

that we use theA std namespace by default: 














#include <iostream> 
#include <optional> 
#include <algorithm> 
#include <functional> 
#include <iterator> 
#include <map> 
#include <vector> 
#include <string> 

using namespace std; 


^ /v 

2. The entireA program revolves around a trieA for which we have to 
implement a class first. In our implementation, a trie is basically a 
recursive map of maps. Every trie node contains a map, which maps 
from an instance of the payload type t to the next trie node: 


template <typename T> 
class trie 
{ 

map<T, trie> tries; 

3. The code for inserting new item sequences is simple. The user provides 
a begin/end iterator pair and we loop through it recursively. If the user 
input sequence is { 1 , 2 , 3 }, then we look up 1 in the subtrie and then 
look up A 2 in the next subtrie, in order to get the subtrie for 3. If any of 
those subtries did not exist before, they are implicitly added by the [ ] 
operator of std: :map: 


public: 

template <typename It> 

void insert (It it. It end_it) { 

if (it == end_it) { return; } 
tries[ *it ].insert(next(it), end_it); 

} 

4. We also define convenience functions, which enable the user to just 
provide a container of items, which are then automatically queried for 
iterators: 

template <typename C> 

void insert(const C Scontainer) { 

insert (begin(container), end(container)); 

} 



/\ 

5. In order to allow the user to writeAmytrie. insert ({"a", "b", 

"c"});, we must help the compiler a bit to correctly deduce all the types 
from that line, so we add a function, which overloads the insert interface 
with an initiaiizer iist parameter: 

void insert(const initializer_list<T> &il) { 
insert (begin(il), end(il)); 

} 

6. We will also want to see what's in a trie, so we need a print function. In 
order to print, we can do a depth-first-search through the trie. On the 
way from the root node down to the first leaf, we record all payload 
items we have seen already. This way, we have a complete sequence 
together once we reach the leaf, which is trivially printable. We see that 

/V 

we reached a leafA when tries. empty () is true. After the recursive 
print call, we pop off the last added payload item again: 

void print(vector<T> &v) const { 
if (tries . empty () ) { 

copy(begin(v), end(v), 

ostream_iterator<T>{cout, " "}); 
cout << 1 \n'; 

} 

for (const auto &p : tries) { 
v.push_back(p.first) ; 
p.second.print(v); 
v.pop_back(); 

} 

} 


7. The recursive print function passes A around a reference to a printable 

/V 

list of payload items, but the user shouldA call it without any 

_ /V 

parameters. Therefore, we define aA parameterless print function, 
which constructs the helper list object: 

void print() const { 
vector<T> v; 
print(v); 

} 

8. Now that we can construct and print tries, we may want to search for 
subtries. The idea is that if the trie contains sequences such asA {a, b. 



/V 

c } and { a , b, d, e}, and we give it a sequence,A { a , b}, for search, it 

/V 

would return us the subtrie that A contains the {c} and {d, e} parts. If 
we find the subtrie, we return a const reference to it. The possibility 
exists that there is no such subtrie in case the trie does not contain the 
sequence we are searching for. In such cases, we still need to return 

_ /V 

something. TheA std:: optional is a nice helper because we can return 
an empty optional object if there is no match: 


template <typename It> 

optionalcreference_wrapper<const trie>> 
subtrie(It it. It end_it) const { 

if (it == end_it) { return ref (*this); } 

auto found (tries.find(*it)); 
if (found == end(tries)) { return {}; } 

return found->second.subtrie(next(it), end_it); 

} 

9. Similar to the insert method, we provide a one-parameter version of 
the subtrie method, which automatically takes iterators from the input 
container: 

template <typename C> 
auto subtrie(const C &c) { 

return subtrie(begin(c), end(c)); 

} 

}; 


10. That's already it. Let's put the new trie class to use in our main function 
by instantiating a trie specialized on std:: string objects and fill it with 
some example content: 


int main() 

{ 

trie<string> t; 

t.insert({"hi", "how", "are", "you"}); 
t.insert({"hi", "i", "am", "great", "thanks"}); 
t.insert({"what" , "are", "you", "doing"}); 
t.insert({"i", "am", "watching", "a", "movie"}); 


11. Let's first print the whole trie: 



cout << "recorded sentences:\n"; 
t.print (); 

12. Then we obtain the sub trie for all the input sentences that start with 
"hi", and print it: 

cout << "\npossible suggestions after \"hi\":\n"; 

if (auto st (t.subtrie(initializer_list<string>{"hi"})) 
st) { 

st->get() .print (); 

} 

} 

13. Compiling and running the program shows that it does indeed return us 

/V 

only the two sentences that A start with "hi", when we query the trie for 
exactly that subtrie: 

$ ./trie 

recorded sentences: 
hi how are you 
hi i am great thanks 
i am watching a movie 
what are you doing 

possible suggestions after "hi": 
how are you 
i am great thanks 


How it works... 


Interestingly, the code for word sequence insertion is shorter and simpler 
than the code for looking up a given word sequence in a subtrie. So, let's first 
have a look at the insertion code: 


template <typename It> 
void trie::insert(It it. It end_it) { 
if (it == end_it) { return; } 
tries[*it].insert(next(it), end it); 


} 



The pair of iterators, A it and end it, represent the word sequence to be 
inserted. The A tries[*it]A element looks up the first word in the sequence 

/V 

in the subtrie, and then,A . insert (next (it), end it) restarts the same 
function on that lower subtrie, with the iterator one word further advanced. 
The if (it == endit) { return; } line just aborts the recursion. The 
empty return statement does nothing, which is a bit weird at first. All the 

insertion happens in the tries [ * it ] statement. The bracket operator [ ] of 

/\ 

std :: map either returns an existing item for the given keyA or it creates one 
with that key. The associated value (the mapped type is a trie in this recipe) is 
constructed from its default constructor. This way, we are implicitly creating 
a new trie branch whenever we are looking up unknown words. 

/V 

Looking up in a subtrie looks more complicatedA because we were not able 
to hide so much in implicit code: 


template ctypename It> 

optional<reference_wrapper<const trie>> 
subtrie(It it. It end_it) const { 

if (it == end_it) { return ref(*this); } 
auto found (tries.find(*it)); 
if (found == end (tries)) { return {}; } 

return found->second.subtrie(next(it), end_it); 

} 

This code basically revolves around theA auto found 

/v 

(tries. find (*it)); A statement. Instead of looking up the next deeper trie 
node using the bracket operator ([]), we use find. If we use the [ ] operator 

/V 

for lookups, the trie willA create missing items for us, which is not what we 

/V 

want when just looking up whetherA an item exists! (By the way, try doing 
that. The class method is const, so this will not even be possible. This can be 
quite a life saver, which helps us in preventing bugs.) 

Another scary looking detail is the return 

/V 

type,A optionalcreference_wrapper<const trie>>. We chose 

/\ /\ 

std: : optional as theA wrapperA because it is possible that there is no such 
subtrie for the input sequence we are looking for. If we only inserted "hello 
my friend", there will be no "goodbye my friend" sequence to lookup. In 
such cases, we just return {}, which gives the caller an empty optional object. 
This still does not explain why we use referencewrapper instead of just 



writing optionai<const trie &>. The point here is that an optional instance 
with a member variable of the tries type is not reassignable and hence 
would not compile. Implementing a reference using reference wrapper 
leads to reassignable objects. 



ImplementingA a search inputA suggestion 
generator with tries 


When entering something into a search engine on the Internet, the interface 

/V 

often tries to guess how the full search queryA will look. This guessing is 

usually based on popular search queries from the past. Sometimes, such 

/\ 

search engine guesses are quite funnyA because it appears that people type 
weird queries into search engines. 


what would| vf/ 

what would 

what would macgyver do 
what would blair waldorf do 
what would beyonce do 

/V 

In this section, we are going to use the trie class thatA we implemented in the 
previous recipe and build a little search query suggestion engine. 


How to do it... 


In this section, we will implement a terminal app, which accepts some 

/\ 

inputA and then tries to guess what the user might want to look for, based on 
a cheap text file database: 

1. As always, includes come first, and we define that we use 

/y /\ 

the A stdA namespace: 

#include <iostream> 

#include <optional> 

#include <algorithm> 

#include <functional> 

#include <iterator> 

#include <map> 

#include <list> 




#include <string> 

#include <sstream> 

#include <fstream> 

using namespace std; 

2. We use the trie implementation from the trie recipe: 

template <typename T> 
class trie 
{ 

map<T, trie> tries; 
public: 

template <typename It> 

void insert (It it. It end_it) { 

if (it == end_it) { return; } 
tries[ *it ] .insert (next(it), end_it); 

} 

template <typename C> 

void insert(const C Scontainer) { 

insert(begin(container), end(container)); 

} 

void insert (const initializer_list<T> &il) { 
insert(begin(il), end(il)); 

} 

void print(list<T> &1) const { 
if (tries . empty () ) { 

copy(begin(1), end(l), 

ostream_iterator<T>{cout, " "}); 
cout << '\n'; 

} 

for (const auto &p : tries) { 

1.push_back(p.first) ; 
p . second.print(1); 

1.pop_back() ; 

} 

} 

void print() const { 
list<T> 1; 
print(1); 


} 



template <typename It> 

optional<reference_wrapper<const trie>> 
subtrie(It it. It end_it) const { 

if (it == end_it) { return ref(*this); } 
auto found (tries.find(*it)); 
if (found == end(tries)) { return {}; } 

return found->second.subtrie(next(it), end_it); 

} 

template <typename C> 

auto subtrie (const C &c) const { 

return subtrie(begin(c) , end(c) ) ; 

} 


3. Let's add a little helper function that prints a line that prompts the user to 
enter some text: 

static void prompt() 

{ 

cout << "Next input please:\n > 

} 

4. In the main function, we open a text file, which acts as our sentence 

/\ 

database. We read that text file line by line A and feed those lines into a 
trie: 


int main() 

{ 

trie<string> t; 
fstream infile {"db.txt"}; 

for (string line; getline(infile, line);) { 
istringstream iss {line}; 

t.insert(istream_iterator<string>{iss}, { }) ; 

} 

5. Now that we have constructed the trie from the content in the text file, 

/V 

we need to implement an interface for the user to query it.A We prompt 
the user to enter some text and wait for a whole line of input: 

prompt(); 

for (string line; getline(cin, line);) { 
istringstream iss {line}; 



6. With that text input, we query the trie in order to get a subtrie from it. If 
we have such an input sequence in the text file already, then we can 
print how the input can be continued, just as in the search engine 
suggestion feature. If we do not find a matching subtrie, we just tell the 
user: 


if (auto st (t.subtrie(istream_iterator<string>{iss 
st) { 

cout << "Suggestions:\n"; 
st->get().print(); 

} else { 

cout << "No suggestions found.\n"; 

} 

7. Afterward, we print the prompt text again and wait for the next line of 
user input. That's it. 

cout << "-\n"; 

prompt(); 

} 

} 

8. Before thinking about launching the program, we need to fill some 
content into db. txt. The input can be really anything, and it does not 
even need to be sorted. Each line of text will be one trie sequence: 

do ghosts exist 

do goldfish sleep 

do guinea pigs bite 

how wrong can you be 

how could trump become president 

how could this happen to me 

how did bruce lee die 

how did you learn C++ 

what would aliens look like 

what would macgiver do 

what would bjarne stroustrup do 


9. We need to create db. txt before we can run the program. Its content 
could look like this: 


hi how are you 




hi i am great thanks 

do ghosts exist 

do goldfish sleep 

do guinea pigs bite 

how wrong can you be 

how could trump become president 

how could this happen to me 

how did bruce lee die 

how did you learn C++ 

what would aliens look like 

what would macgiver do 

what would bjarne stroustrup do 

what would chuck norris do 

why do cats like boxes 

why does it rain 

why is the sky blue 

why do cats hate water 

why do cats hate dogs 

why is C++ so hard 


10. Compiling and running the program and entering some input looks like 
the following: 

$ ./word_suggestion 
Next input please: 

> what would 
Suggestions: 
aliens look like 
bjarne stroustrup do 
chuck norris do 
macgiver do 


Next input please: 

> why do 
Suggestions: 
cats hate dogs 
cats hate water 
cats like boxes 


Next input please: 
> 


How it works... 


How a trie works was explained in the last recipe, but how we fill it and how 





we query it looks a bit strange here. Let's have a closer look at the code 

/V 

snippet thatA fills the empty trie with the content of the text database file: 

fstream infile {"db.txt"}; 

for (string line; getline(infile, line);) { 
istringstream iss {line}; 

t. insert (istream_ iterator<string> { iss }, { }) ; 

} 

The loop fills the string line with the content of the text file, line by line. 

Then, we copy the string into an istringstream object. From such an input 

/\ 

stream object, we can create an istream iterator, whichA is useful because 

/y 

our trie does not only accept a container instance for looking up subtriesA but 
also primarily iterators. This way, we do not need to construct a vector or a 
list of words and can directly consume the string. The last piece of 

unnecessary memory allocations could be avoided by moving the content of 

/\ 

line intoA iss. Unfortunately, std:: istringstream does not provide a 

/v /y /y 

constructor thatA accepts std:: stringA valuesA to be moved. It will copy 
its input string, nevertheless. 

When reading the user's input to look it up in the trie, we use exactly the 

/y 

same strategy but we do not use an input file stream. We useA std:: cin, 
instead. This works completely identically for our use caseA because 
trie : : subtrie works with iterators just as trie : : insert does. 


There’s more... 


It is possible to add counter variables to each node of the trie. This way, it is 

/V 

possible to count how often A a prefix occurs in some input. From that, we 
could sort our suggestions by their occurrence frequency, which is actually 
what search engines do. Word suggestions for smartphone touchscreen text 
input could also be implemented this way. 


/v 

This modification is left as an exercise for the reader.A 



Implementing the Fourier transform 
formula with STL numeric algorithms 


The Fourier transformation is a very important and famous formula in 
signal processing. It was invented nearly 200 years ago, but with computers, 
the number of use cases for it really skyrocketed. It is used in 
audio/image/video compression, audio filters, medical imaging devices, cell 

/V 

phone apps thatA identify music tracks while listening to them on the fly, and 
so on. 

Because of the vastness of general numeric application scenarios (not only 
because of the Fourier transformation of course), the STL also tries to be 
useful in the context of numeric computation. The Fourier transformation is 
only one example amongA them but a tricky one too. The formula itself 
looks like the following: 



The transformation it describes is basically a sum. Each element of the sum is 
the multiplication of a data point of the input signal vector, and the 
expressionA exp(-2 * i *...). The maths behind this is a bit scary for everyone 
who does not know about complex numbers (or who just does not like 
maths), but it is also not really necessary to completely understand the maths 
in order to implement it. When having a close look at the formula, it says that 
the sum symbol loops over every data point of the signal (which is n elements 
long) using the loop variable j . The variable k is another loop 

/V 

variableA because the Fourier transformation is not for calculating a single 
value, but a vector of values. In this vector, every data point represents the 




intensity and phase of a certain repetitive wave frequency, which is or is not a 
part of the original signal. When implementing this with manual loops, we 
will end up with code similar to the following: 


csignal fourier_transform(const csignal &s) { 

csignal t(s.size() ) ; 

const double pol {-2.0 * M_PI / s.sizef)}; 

for (size_t k {0}; k < s.size(); ++k) { 

for (size_t j {0}; j < s.size(); ++j) { 

t[k] += s[j] * polar(1.0, pol * k * j); 

} 

} 

return t; 

} 

/v 

The csignal type may be an std: : vectorA vector of complex numbers. For 
complex numbers, there is anA std: : complex STL class, which helps 
represent those. The std: : polar function basically does the exp(-i * 2 * ...) 
part. 

This works well already, but we are going to implement it using STL tools. 


How to do it... 


In this section, we are going to implement the Fourier transformation and its 
backward transformation and then play around with it to transform some 
signals: 

1. First, we include all the headers and declare that we use 

/\ /V 

theA stdA namespace: 

#include <iostream> 

#include <complex> 

#include <vector> 

#include <algorithm> 

#include <iterator> 

#include <numeric> 

#include <valarray> 

#include <cmath> 


using namespace std; 



2. A data point of a signal is a complex number and shall be represented by 

/V /V 

std:: complex, specialized on theA doubieA type. This way, the type 
alias cmpix stands for two coupled double values, which represent the 
real and the imaginary parts of a complex number. A whole signal is a 

/v 

vector of such items, which we alias to theA csignaiA type: 

using cmpix = complex<double>; 
using csignal = vector<cmplx>; 


3. In order to iterate over an up-counting numeric sequence, we take the 

_ /\ 

numeric iterator from the numeric iterator recipe. The variables kA and 
j in the formula shall iterate over such sequences: 

class num_iterator { 
siz e_t i; 
public: 

explicit num_iterator(size_t position) : i{position} {} 
size_t operator*() const { return i; } 
num_iterator& operator++() { 

_ ++i; 

return *this; 

} 

bool operator!=(const num_iterator Sother) const { 
return i != other.i; 

} 

}; 


4. The Fourier transformation function shall just take a signal and return a 
new signal. The returned signal represents the Fourier transformation of 
the input signal. As the back transformation from a Fourier transformed 
signal back to the original signal is very similar, we provide an optional 
bool parameter, which chooses the transformation direction. Note that 
bool parameters are generally bad practice, especially if we use multiple 
bool parameters in a function signature. Here we just have one for 
brevity. The first thing we do is allocate a new signal vector with the 
size of the initial signal: 

csignal fourier_transform(const csignal &s, bool back = 

{ 


fal 



csignal t (s.size ()); 


5. There are two factors in the formula, which always look the same. Let's 
pack them in their own variables: 

const double pol {2.0 * M_PI * (back ? -1.0 : 1.0)}; 
const double div {back ? 1.0 : double(s.size())}; 

6. The std:: accumulate algorithm is a fitting choice for executing 
formulas that sum up items. We are going to use accumulate on a range 
of up-counting numeric values. From these values, we can form the 

_ /V /V 

individual summands of each step. TheA std:: accumuiateA algorithm 

calls a binary function on every step. The first parameter of this function 

/\ 

is the current value of the part of sumA that was already calculated in the 
previous steps, and its second parameter is the next value from the 

/V /V /V 

range. We look up the value ofA signalA s at the current positionA and 
multiply it with the complex factor,A pol. Then, we return the new 
partly sum. The binary function is wrapped into another lambda 

/V 

expressionA because we are going to use different values of j for 

/V 

everyA accumulate call. Because this is a two-dimensional loop 
algorithm, the inner lambda is for the inner loop and the outer lambda is 
for the outer loop: 


auto sum_up ([=, &s] (size_t j) { 

return [=, &s] (cmplx c, size_t k) { 

return c + s[k] * 

polar(1.0, pol * k * j / double(s.size())); 

} ; 

}) ; 

7. The inner loop part of the Fourier transform is now executed by 
std: : accumulate. For every j position of the algorithm, we calculate 
the sum of all the summands for positions i = 0...N. This idea is wrapped 
into a lambda expression, which we will execute for every data point in 
the resulting Fourier transformation vector: 

auto to_ft ([=, &s](size_t j){ 

return accumulate(num^iterator{0}, 

num_iterator{s.size() }, 
cmplx{}, 
sum up (j)) 



/ div; 


}) ; 

A 

8. None of the Fourier code has beenA executed until this point. We only 
prepared a lot of functional code, which we'll put to action now. An 
std:: transform call will generate values j = 0...N, which is our outer 
loop. The transformed values all go to the vector t, which we then return 
to the caller: 


transform(num_iterator{0}, num_iterator{s.size() }, 
begin(t), to_ft); 


return t; 

} 

9. We are going to implement some functions that help us set up function 

_ /\ 

objects for signal generation. The first one is aA cosine signal generator. 

/\ 

It returns a lambda expression thatA can generate a cosine signal with 

/\ _ 

the period length thatA was provided as a parameter. The signal itself 
can be of arbitrary length, but it has a fixed period length. A period 
length of n means that the signal will repeat itself after n steps. The 
lambda expression does not accept any parameters. We can call it 
repeatedly, and for every call, it returns us the signal data point of the 
next point in time: 


static auto gen_cosine (size_t period_len){ 
return [period_len, n{0}] () mutable { 

return cos(double(n++) * 2.0 * M_PI / period_len); 

} ; 

} 

/\ 

10. Another signal we are going to generateA is the square wave. It 

/V 

oscillates between the values -1 and + iA and has no other values than 
those. The formula looks complicated, but it simply transforms the 
linearly up-counting value n to +i and -l, with an oscillating period 
length of period ien. Note that we initialize n to a different value 

/V ~ 

fromA o this time. This way, our square wave starts at the phase where 
its output values begin at +i: 


static auto gen_square_wave (size_t period_len) 
{ 



return [period_len, n{period_len*7/4}] () mutable { 

return ( (n++ * 2 / period_len) % 2) * 2 - 1.0; 

}; 

} 

11. Generating an actual signal from such generators can be achieved by 
allocating a new vector and filling it with the values generated from 
repeating signal generator function calls. TheA std: generate does this 
job. It accepts a begin/end iterator pair and a generator function. For 
every valid iterator position, it does *it = gen (). By wrapping this 
code into a function, we can easily generate signal vectors: 


template <typename F> 

static csignal signal_from_generator(size_t len, F gen) 

{ ' ’ “ 
csignal r (len); 

generate(begin (r), end(r), gen); 
return r; 

} 

12. In the end, we need to print the resulting signals. We can simply print a 

/V 

signal by copying its values into an output stream iterator,A but we need 
to transform the data first because the data points of our signals are 
complex value pairs. At this point, we are only interested in the real 

value part of every data point; hence, we throw it through 

/\ 

anA std:: transform call, which extracts only this part: 

static void print_signal (const csignal &s) 

{ 

auto real_val ([](cmplx c) { return c.real(); }); 

transform(begin(s), end(s), 

ostream_iterator<double>{cout, " "}, real_val 
cout << 1 \n 1 ; 

} 

13. The Fourier formula is now implemented, but we have no signals to 
transform yet. That is what we do in the main function. Let's first define 
a standard signal length to which all the signals comply. 


int main() 
{ 


const size t sig len {100}; 



/V 

14. Let's nowA generate signals, transform them, and print them, which 
happens in the next three steps. The first step is to generate a cosine 
signal and a square wave signal. Both have the same total signal length 
and period length: 


auto cosine (signal_from_generator(sig_len, 

gen_cosine( sig_len / 2))); 

auto square_wave (signal_from_generator(sig_len, 
gen_square_wave(sig_len / 2))); 

15. We have a cosine function and a square wave signal now. In order to 

generate a third one in the middle between them, we take the square 

/\ 

wave signalA and calculate its Fourier transform (saved in 

/\ /\ _ 

theA trans sqwA vector). The Fourier transform of a square wave has a 

specific form, and we are going to manipulate it a bit. All items from 

/\ 

indexA 10 till (signai iength - 10) are set to o. o. The rest remains 
untouched. Transforming this altered Fourier transformation back to the 
signal time representation will give us a different signal. We will see 
how that looks in the end: 


auto trans_sqw (fourier_transform(square_wave)); 

fill (next(begin(trans_sqw), 10), prev(end(trans_sqw), 

auto mid (fourier_transform(trans_sqw, true)); 

16. Now we have three signals: cosine, mid, and square wave. For every 
signal, we print the signal itself and its Fourier transformation. The 
output of the whole program will consist of six very long lines of printed 
double value lists: 

print_signal(cosine); 

print_signal(fourier_transform(cosine)); 

print_signal(mid); 

print_signal(trans_sqw); 

print_signal(square wave); 

print_signal(fourier_transform(square_wave)); 

} 

17. Compiling and running the program leads to the terminal getting filled 
with lots of numeric values. If we plot the output, we get the following 



image: 



How it works... 

This program contains two complicated sections. One is the Fourier 
transformation itself, and the other is the generation of signals with mutable 
lambda expressions. 

Let's concentrate on the Fourier transformation first. The core of the raw loop 
implementation (which we did not use for our implementation but had a look 
at in the introduction) looks like the following: 

for (size_t k {0}; k < s.size (); ++k) { 

for (size_t j {0}; j < s.size (); + +j) { 

t[k] += s[j] * polar(1.0, pol * k * j / double(s.size())) 

} 

} 

/\ _ /\ 

WithA the STL algorithms,A std::transform and std::accumulate, we 
wrote code, which can be summarized to the following pseudo code: 









































transform(num_iterator{0}, num_iterator{s.size() }, ... 

accumulate((num_iteratorO}, num_iterator{s.size()}, ... 

c + s[k] * polar(1.0, pol * k * j / double(s.size())); 

The result is exactly the same compared withA the loop variant. This is 
arguably an example situation where the strict use of STL algorithms does 
not lead to better code. Nevertheless, this algorithm implementation is 
agnostic over the data structure choice. It would also work on lists (although 
that would not make too much sense in our situation). Another upside is that 
the C++17 STL algorithms are easy to parallelize (which we examine in 
another chapter of this book), whereas rawA loops have to be restructured to 
support multiprocessing (unless we use external libraries like OpenMP for 

/V 

example,A but these do actually restructure the loops for us). 

The other complicated part was the signal generation. Let's have another look 

at gen_cosine! 

static auto gen_cosine (size_t period_len) 

{ 

return [period_len, n{0}] () mutable { 

return cos(double(n++) * 2.0 * M_PI / period_len); 

}; 

} 

Each instance of the lambda expression represents a function object 

/V 

that A modifies its own state on every call. Its state consists of the 

/\ _ 

variables,A period ien and n. The n variable is the one which is modified on 
every call. The signal has a different value at every time point, and n++ 
represents the increasing time points. In order to get an actual signal vector 
out Of it, we created the helperA signal_from_generator: 


template <typename F> 

static auto signal_from_generator(size_t len, F gen) 
{ 

csignal r (len); 

generate(begin(r), end(r), gen); 
return r; 

} 


This helper allocates a signal vector with a length of choiceA and calls 
std:: generate to fill it with data points. For every item of the vector r, it 



calls the function object gen once, which is just the kind of self-modifying 
function object we can create with gen cosine. 

Note 

Unfortunately, the STL way does not make this code more 
elegant. As soon as the ranges library joins the STL club (which 
is hopefully the case with C++20), this will most probably 
change. 



Calculating the error sum of two vectors 


/V /V 

There are different possibilities to calculate the numericalA error A between a 
target value and an actual value. Measuring the difference between signals 
consisting of many data points usually involves loops and subtraction of 
corresponding data points, and so on. 

One simple formula to calculate this error between a signal a and a signal b is 
the following: 

e = Y J { a ,-b l ) 1 

i =0 

For every i, it calculates a[i] - b[i], squares that difference (this way, negative 
and positive differences become comparable), and, finally, sums those values 
up. This is again a situation where one could use a loop, but for fun reasons, 
we will do itA with an STL algorithm. The good thing is that we get data- 
structure independence for free this way. Our algorithm will work on vectors 
and on list-like data structures, where no direct indexing is possible. 

How to do it... 

In this section, we are going to create two signals and calculate their error 
sum: 

1. As always, the include statements come first. Then, we declare that we 

/\ 

use the stdA namespace: 


#include <iostream> 
#include <cmath> 




#include <algorithm> 
#include <numeric> 

#include <vector> 
#include <iterator> 

using namespace std; 


2. We are going to calculate the error sum of two signals. The two signals 

/V /\ 

will be aA sine waveA and a copy of it, but with a different value type— 
the original sine wave is saved in a vector of double variables and its 
copy is saved in a vector of int variables. Because copying a value from 
a double variable to an int variable cuts its decimal part after the point, 
we have some loss. Let's name the vector of double values as, which 

/V 

stands forA analog signal and the vector of int values ds, which stands 

/\ _ 

forA digital signal. The error sum will then later tell us how large the 
loss actually is: 


int main() 

{ 

const size_t sig_len {100} ; 

vector<double> as (sig_len); // a for analog 
vector<int> ds (sig_len); // d for digital 

3. In order to generate a sine wave signal, we implement a little lambda 

/\ 

expression with a mutable counter valueA n. We can call it as often as 
we want, and for every call, it will return us the value for the next point 
in time of a sine wave. The std:: generate call fills the signal vector 
with the generated signal, and the std:: copy call copies all the values 
from the vector of double variables to the vector of int variables 
afterward: 


auto sin_gen ([n{0}] () mutable { 

return 5.0 * sin(n++ * 2.0 * M_PI / 100); 

}); 

generate(begin(as), end(as), sin_gen); 
copy(begin(as), end(as), begin(ds)); 


4. Let's first print the signals, as this way, they can be plotted later: 



copy(begin(as), end(as), 

ostream_iterator<double>{cout, " "}); 
cout << ' \n' ; 
copy(begin(ds), end(ds), 

ostream_iterator<double>{cout, " "}); 
cout << '\n'; 

/\ 

5. Now to the actual error sum, we use std:: inner productA because it 
can easily be adapted to calculate the difference between every two 
corresponding elements of our signal vectors. It will iterate through both 
the ranges, pick items at the same corresponding positions in the ranges, 

/V 

calculate the difference betweenA them, square it, and accumulate the 
results: 


cout << inner_product(begin(as), end(as), begin(ds), 

0.0, std::plus<double>{}, 

[](double a, double b) { 
return pow(a - b, 2); 

}) 

« ' \ n ' ; 

} 

6. Compiling and running the program gives us two long lines of signal 

/V 

outputA and a third line, which contains a single output value, which is 
the error between both the signals. The error is 4 o. 8 8 9. If we calculate 
the error in a continuous manner, first for the first pair of items, then for 

/\ /V 

the first twoA pairs of items, then for the first threeA pairs of items, and 
so on, then we get the accumulated error curve, which is visible on the 
plotted graph as shown: 




0 1 0 2 0 30 4 0 50 6 0 70 8 0 90 1 00 



How it works... 


In this recipe, we stuffed the task of looping through two vectors, getting the 

/V 

difference betweenA their corresponding values, squaring them, and finally 

summing them up into one std:: inner product call. On the way, the only 

/\ 

code we crafted ourselves wasA the lambda expression [ ] (double a, 
double b) { return pow(a - b, 2); }, which takes the difference of its 
arguments and squares it. 

A glance at a possible implementation of std:: inner product shows us why 
and how this works: 

template<class Inltl, class Inlt2, class T, class F, class G> 

T inner_product(Initl itl, Inltl endl, Inlt2 it2, T val, 

F bin_opl, G bin_op2) 

{ 

while(itl!= endl){ 

val = bin_opl(val, bin_op2(*itl, *it2)); 

++itl; 

++it2; 

} 

return value; 
























} 


/v 

The algorithm accepts a pair of begin/end iterators of theA first range, and 

/\ 

another begin iterator of theA second range. In our case, they are the vectors 
from which we want to calculate the error sum. The next character is the 

/V ^ 

initial value A vai. We have initialized it to o. o. Then, the algorithm accepts 
two binary functions, namely bin opi and bin_op2. 

At this point, we might realize that this algorithm is really similar to 
std:: accumulate. The only difference is that std: accumulate works on 
only one range. If we exchange the bin_op 2 (*iti, *it2 ) statement with 
*it, then we have basically restored the accumulate algorithm. We can, 
therefore, regard std:: inner product as a version of 

/V /V 

std:: accumuiateA thatA zips a pair of input ranges. 

In our case, the zipper function is pow (a - b, 2 ), and that's it. For 
theA other function, bin opi, we chose std: :pius<doubie> because we 
want all the squares to be summed together. 



Implementing an ASCII Mandelbrot 
renderer 


In 1975, the mathematician BenoA®t Mandelbrot coined the term fractal. A 
fractal is a mathematical figure or set, which has certain interesting 
mathematical properties, but in the end, it just looks like a piece of art. 

/V 

Fractals also look infinitely A repetitive when being zoomed in. One of the 
most popular fractals is the Mandelbrot set, which can be seen on the 
following poster: 



/V 

AA picture of the Mandelbrot set can be generated by iterating a specific 
formula: 








/V /\ /V 

The variables A zA and cA are complex numbers. The Mandelbrot set consists 

/\ /V 

of all such values ofA cA for which the formula converges if it is applied 
often enough. This is the colored part of the poster. Some values converge 
earlier, some converge later, so they can be visualized with different colors. 
Some do not converge at all—these are painted black. 

The STL comes with the useful std:: complex class, and we will try to 
implement the formula without explicit loops, just for the sake of getting to 
know the STL better. 


How to do it... 


/\ 

In this section, A we are going to print the same image from the wall poster as 
a little piece of ASCII art in the terminal: 

1. First, we include all the headers and declare that we use 

/\ /V 

theA stdA namespace: 

#include <iostream> 

#include <algorithm> 

#include <iterator> 

#include <complex> 

#include <numeric> 

#include <vector> 

using namespace std; 


2. The Mandelbrot set and formula operate on complex numbers. So, we 

/V 

define a type alias,A cmpix to be of class std:: complex, specializing on 
double values. 

using cmpix = complex<double>; 



3. It is possible to hack together all the code for an ASCII Mandelbrot 
image in something around 20 lines of code, but we will implement each 
logical step in a separate form, and then assemble all the steps in the 
end. The first step is implementing a function thatA scales from integer 
coordinates to floating point coordinates. What we have in the beginning 
is columns and rows of character positions on the terminal. What we 
want are complex-typed coordinates in the coordinate system of the 

/V /V 

Mandelbrot set. For this, we implement aA function thatA accepts 
parameters that describe the geometry of the user terminal coordinate 
system, and the system we want to transform to. Those values are used 
to build a lambda expression, which is returned. The lambda expression 
accepts an int coordinate and returns a double coordinate: 


static auto 
{ 


scaler(int min_from, int max_from, 

double min to, double max to) 


const int w_from (max_from - min_from}; 

const double w_to (max_to - min_to}; 

const int mid_from {(max_from - min_from) / 2 + min_ 

const double mid to {(max to - min to) / 2.0 + min to 


return [=] (int from) { 

return double(from - mid_from) / w_from * w_to + mi 

}; 


4. Now we can transform points on one dimension, but the Mandelbrot set 
exists in a two-dimensional coordinate system. In order to translate from 
one (x, y) coordinate system to another, we combine an x-scaler and a 

/V 

y-scalerA and construct a cmpix instance from their output: 

template <typename A, typename B> 

static auto scaled_cmplx(A scaler_x, B scaler_y) 

{ 

return [=](int x, int y) { 

return cmpix{scaler_x(x), scaler_y(y) } ; 

}; 

} 


5. After being able to transform coordinates to the right dimensions, we 

_ /\ 

can now implement the Mandelbrot formula. The function thatA we're 

/V 

implementing nowA knows absolutely nothing about the concept of 



/\ /V 

terminalA windowsA or linear plane transformations, so we can 

/\ /\ 

concentrate on the Mandelbrot math. We square z and addA cA to it in a 

/V 

loopA until its abs value is smaller than 2 . For some coordinates, this 
never happens, so we also break out of the loop if the number of 
iterations exceeds max iterations. In the end, we return the number of 

/V 

iterations we had to do A until the abs value converged: 


static auto mandelbrot_iterations(cmplx c) 

{ 

cmplx z {}; 

size_t iterations {0}; 

const size_t max_iterations {1000}; 

while (abs(z) < 2 && iterations < max_iterations) { 

++iterations; 
z = pow(z, 2) + c; 

} 

return iterations; 

} 

6. We can now begin with the main function, where we define the terminal 

/V /\ 

dimensions and instantiate a function object,A scale, whichA scales our 
coordinate values for both axes: 


int main() 

{ 

const size_t w {100} ; 
const size t h {40}; 


auto scale (scaled_cmplx( 

scaler(0, w, -2.0, 1.0), 
scaler(0, h, -1.0, 1.0) 


7. In order to have a one-dimensional iteration over the whole image, we 

/V 

write another transformation function thatA accepts a one-dimensional 

/\ 

iA coordinate. It calculates (x, y) coordinates from that, based on our 
assumed line of characters width. After breaking i down to the row and 
column numbers, it transforms them with our scale function and returns 
the complex coordinate. 


auto i to xy ([ = ] (int i) { return scale (i % w, i / w); 



8. What we can do now is transform from one-dimensional coordinates 

/V /V /V 

(theA intA type), via two-dimensional coordinates (theA (int, 

/V /\ /\ 

int) A type), to Mandelbrot set coordinates (theA cmpixA type), and 

/\ /V 

then calculate the number of iterations from there (theA intA type 
again). Let's combine all that in one function, which sets up this call 
chain for us: 


auto to_iteration_count ([=](int i) { 

return mandelbrot_iterations(i_to_xy(i)); 

}) ; 

9. Now we can set up all the data. We assume that our resulting ASCII 
image is w characters wide and h characters high. This can be saved in a 
one-dimensional vector that has w * h elements. We fill this vector 

/V 

using std :: iota with the value range,A 0 ... (w*h -1). These numbers 
can be used as an input source for our constructed transformation 
function range, which we just encapsulated in to_iteration_count: 

vector<int> v (w * h); 
iota(begin(v), end(v), 0); 

transform(begin(v), end(v), begin(v), to_iteration_coun 

_ /v 

10. That's basically it. We now have theA v vector, which we initialized 
with one-dimensional coordinates, but which then got overwritten by 
Mandelbrot iteration counters. From this, we can now print a pretty 
image. We could just make the terminal window w characters wide, then 
we would not need to print line break symbols in between. But we can 

also kind of creatively misuse std: : accumulate to do the line breaks for 

/v 

us. TheA std: : accumulate uses a binary function to reduce a range. We 
provide it a binary function, which accepts an output iterator (and which 
we will link to the terminal in the next step), and a single value from the 

/V 

range. We print this value as a * A character if the number of iterations is 
higher than 50. Otherwise, we just print a space character. If we are on a 
row end (because the counter variable n is evenly divisible by w), we 
print a line break symbol: 


auto binfunc ([w, n{0}] (auto output^it, int x) mutable 
*++output_it = (x > 50 ? ' * ' : ' 1 ) ; 

if (++n % w == 0) { ++output_it = 1 \n 1 ; } 

return output it; 



11. By calling std : accumulate on the input range, combined with our 
binary print function and an ostream iterator, we can flush the 
calculated Mandelbrot set out to the terminal window: 


} 


accumulate(begin(v) , end(v), 
binfunc); 


ostream iterator<char>{cou 


12. Compiling and running the program leads to the following output, which 
looks like the initial detailed Mandelbrot image, but in a simplified 
form: 


$ ./mandelbrot 


** 

******* 

******** 

****** 

* ******** 

*** ******************* 
************************* *** 
***************************** 

** ***************************** 
******************************** 
************************************* 
* *********************************** 

** ****** ************************************ 

*********** ************************************* 

**************** ************************************* 
**************** ************************************* 
*** **************************************************** 
*************************************************************************** 

*** **************************************************** 
**************** ************************************* 
**************** ************************************* 
*********** ************************************* 

** ****** ************************************ 

* *********************************** 

************************************* 
******************************** 

** ***************************** 
***************************** 
************************* *** 

*** ******************* 

* ******** 

****** 

******** 

******* 

** 


How it works... 



The whole calculation took part during an std:: transform call over a one¬ 
dimensional array: 


vector<int> v (w * h); 
iota(begin(v), end(v), 0); 

transform(begin(v), end(v), begin(v), to_iteration_count); 

So, what exactly happened, and why does it work this way? The 
to iteration count function is basically a call chain from i to xy, over 
scale to mandeibrobliterations. The following diagram illustrates the 
transformation steps: 


i_to_xy() 




This way, we can use the index of a one-dimensional array as input, and get 
the number of Mandelbrot formula iterations at the point of the two- 

dimensional plane, which this array point represents. The good thing is that 

/\ 

these threeA transformations are completely agnostic about each other. Code 
with such a separation of concerns can be tested very nicely because each 

/V 

component can be tested individuallyA without the others. This way, it is 
easy to find and fix bugs, or just reason about its correctness. 




Building our own algorithm - split 


In some situations, the existing STL algorithms are not enough. But nothing 
hinders us from implementing our own. Before solving a specific problem, 
we should think about it firmly in order to realize that many problems can be 
solved in generic ways. If we regularly pile up some new library code while 

solving our own problems, then we are also helping our fellow programmers 

/\ 

when they have similar problems to solve.A Key is to know when it is 

/V 

generic enough and when not to go forA more genericity than needed—else 
we end up with a new general purpose language. 

In this recipe, we are implementing an algorithm, which we will call split. It 

can split any range of items at each occurrence of a specific value, and it 

/\ 

copies the chunks that A result from that into an output range. 

How to do it... 


In this section, we are going to implement our own STL-like algorithm called 

/V 

split, and then we checkA it out by splitting an example string: 

/V 

1. First things first,A we include some STL library parts and declare that 

/V 

we use the stdA namespace: 

#include <iostream> 

#include <string> 

#include <algorithm> 

#include <iterator> 

#include <list> 

using namespace std; 


2. The whole algorithm this section revolves around is split. It accepts a 
begin/end pair of input iterators, and an output iterator, which makes it 
similar to std: : copy or std: : transformA at first. The other parameters 

/V _ /V /V 

arc split vs. lA and bin_func. The A spi it vaiA parameter is the 




value we are searching for in the input range, which represents a 

splitting point at which we cut the input interval. 

_ /\ /\ /\ /\ 

TheA bin funcA parameter is a function thatA transforms a pairA of 

/\ 

iterators thatA mark the beginning and the end of such a split chunk 
subrange. We iterate through the input range using std:: find, so we 
jump from occurrence to occurrence of spiit vai values. When 

/V 

splitting a long string into its individualA words, we would jump from 
space character to space character. On every split value, we stop by to 
form a chunk and feed it into the output range: 


template <typename Inlt, 
Inlt split(Inlt it, Inlt 
F bin func) 


{ 


typename Outlt, typename T, typena 
end_it, Outlt out_it, T split_val. 


while (it != end_it) { 

auto slice_end (find(it, end_it, split_val)); 
*out it++ = bin func(it, slice end); 


if (slice_end == end_it) { return end_it; } 
it = next(slice_end); 

} 

return it; 

} 


/v 

3. Let's use the new algorithm. We construct a string thatA we want to 
split. The item thatA marks the end of the last chunk, and 

/V /V 

theA beginning of the next chunk, shall be the dash characterA 1 - 1 : 


int main() 

{ 

const string s {"a-b-c-d-e-f-g"}; 

4. Whenever the algorithm calls its bin func on a pair of iterators, we 
want to construct a new string from it: 


auto binfunc ([](auto it_a, auto it_b) { 
return string(it_a, it_b); 

}) ; 

/v 

5. The output range willA be an std:: list of strings. We can now call the 
split algorithm, which has a similar design compared to all the other 
STL algorithms: 



list<string> 1; 

split(begin(s) , end(s), back_inserter(1) , binfunc) 

6. In order to see what we got, let's print the new chunked list of strings: 

copy(begin(1), end(l), ostream_iterator<string>{cout, " 


7. Compiling and running the program yields the following output. It 

/\ 

contains no dashes anymoreA and shows that it has isolated the 
individual words (which are, of course, only single characters in our 
example string): 

$ ./split 

a 

b 

c 

d 

e 

f 

g 


How it works... 


The split algorithm works in a similar manner to std:: transformA because 
it accepts a pair of begin/end iterators of an input range and an output iterator. 
It does something with the input range, which, in the end, results in 

assignments to the output iterator. Apart from that, it accepts an item value 

/\ 

called spiit vaiA and a binary function. Let's revisit the whole 
implementation to fully understand it: 


template <typename Inlt, typename Outlt, typename T, typename F> 
Inlt split(Inlt it, Inlt end_it, Outlt outfit, T split_val, F bin 
{ 

while (it != end_it) { 

auto slice_end (find(it, end_it, split_val)); 

*out_it++ = bin_func(it, slice_end); 

if (slice_end == end_it) { return end_it; } 
it = next(slice end); 



} 

return it; 

} 

The loop demands to iterate until the end of the input range. During each 
iteration, an std:: find call is used to find the next element in the input 
range, which equals to spiitvai. In our case, that element is the dash 
character (' -') because we want to split our input string at all the dash 
positions. The next dash position is now saved in siice end. After the loop 

/V 

iteration, theA it iterator is put on the next item past that split position. This 
way, the loop jumps directly from dash to dash, instead of over every 
individual item. 

In this constellation, the iterator it points to the beginning of the last slice, 

while siiceend points to the end of the last slice. Both these iterators, in 

/\ 

combination, mark the beginning and end of the subrange thatA represents 

/V 

exactly one slice between two dash symbols. In a string,A "foo-bar-baz", 

/\ 

this would mean that we have threeA loop iterations and we get a pair of 
iterators every time, which surround one word. But we do not actually want 

/V /V 

iterators but substrings. The binary function,A bin func, does justA that 
for us. When we called split, we gave it the following binary function: 

[](auto it_a, auto it_b) { 

return string(it_a, it_b); 

} 

/v 

The spiitA function throws every pair of iterators through binfunc, before 

/\ 

feeding it into the output iterator. AndA we actually get string instances out 
of bin func, which results in "foo", "bar", and "baz": 

There’s more... 

An interesting alternative to implementing our own algorithm for splitting 
strings would be implementing an iterator that does the same. We are not 
going to implement such an iterator at this point, but let's have a brief look at 
such a scenario. 

The iterator would need to jump between delimiters on every increment. 



Whenever it is dereferenced, it needs to create a string object from the iterator 

positions it currently points to, which it could do using a binary function such 

/\ 

asA binfunc, which we used before. 

If we had an iterator class called splitsiterator, instead of an algorithm 
split, the user code would look as follows: 

string s {"a-b-c-d-e-f-g"}; 
list<string> 1; 

auto binfunc ([] (auto it_a, auto it_b) { 
return string(it_a, it_b); 

}); 

copy(split_iterator{begin(s), end(s), a-a, binfunc},{}, back_inse 

The downside of this approach is that implementing an iterator is usually 
more complicated than a single function. Also, there are many subtle 

/V 

edgesA in iterator code that can lead to bugs, so an iterator solution needs 
more tedious testing. On the other hand, it is very simple to combine such an 
iterator with the other STL algorithms. 



Composing useful algorithms from 
standard algorithms - gather 


A very nice example for the composability of STL algorithms is gather. 

Sean Parent, principal scientist at Adobe Systems at the time, popularized this 
algorithm because it is both useful and short. The way it is implemented, it is 
the ideal poster child for the idea of STL algorithm composition. 

The gather algorithm operates on ranges of arbitrary item types. It modifies 
the order of the items in such a way that specific items are gathered around a 
specific position, chosen by the caller. 

How to do it... 


In this section, we will implement the gather algorithm and a bonus variation 
of it. Afterward, we see how it can be put to use: 

1. First, we add all the STL include statements. Then, we declare that we 

/V /\ 

use theA stdA namespace: 

#include <iostream> 

#include <algorithm> 

#include <string> 

#include <functional> 

using namespace std; 


2. The gather algorithm is a nice example of standard algorithm 
composition, gather accepts a begin/end iterator pair, and another 
iterator gather pos, which points somewhere in between. The last 
parameter is a predicate function. Using this predicate function, the 

/V 

algorithm will push all that items thatA do satisfy the predicate near the 
gather pos iterator. The implementation of the item movement is done 
by std:: stabiepartition. The return value of the gather algorithm is 




a pair of iterators. These iterators are returned from the 

stabie partition calls, and this way, they mark the beginning and the 

end of the now gathered range: 

template <typename It, typename F> 

pairCIt, It> gather(It first. It last. It gather_pos, F pre 

{ 

return {stable_partition(first, gather_pos, not_fn(pred 
stable_partition(gather_pos, last, predicate)}; 

} 

3. Another variant of gather is gather sort. It basically works the same 

/V 

way as gather, but it does not accept a unary predicate function;A it 
accepts a binary comparison function instead. This way, it is possible to 
gather the values near gather pos, which appear smallest or largest: 

template ctypename It, typename F> 

void gather_sort(It first. It last. It gather_pos, F comp_f 

{ 

auto inv_comp_func ([&](const auto &...ps) { 

return !comp_func(ps...); 

}); 

stable_sort(first, gather_pos, inv_comp_func); 

stable_sort(gather_pos, last, comp_func); 

} 

4. Let's put those algorithms to use. We start with a predicate, which tells if 
a given character argument is the ’ a ’ character. We construct a string, 
which consists of wildly interleaved 'a' and characters: 


int main() 

{ 

auto is_a ([](char c) { return c == 'a'; }); 

string a {"a_a_a_a_a_a_a_a_a_a_a"}; 

/v 

5. We construct anA iterator, which points to the middle of our new string. 
Let's call gather on it and see what happens. The ' a ' characters should 
be gathered around the middle afterward: 

auto middle (begin(a) + a.size() / 2); 

gather(begin(a), end(a), middle, is_a); 
cout << a << '\n'; 




6. Let's call gather again, but this time, the gather pos iterator is not in 
the middle but the beginning: 


gather(begin(a), end(a), begin (a), is_a); 
cout << a << 1 \n 1 ; 

7. In a third call, we gather items around the end iterator: 

gather(begin(a), end(a), end(a), is_a); 
cout << a << '\n'; 

/\ 

8. With a last call ofA gather, we try to gather all the ' a ' characters 
around the middle again. This will not work as expected, and we will 
later see why: 


// This will NOT work as naively expected 
gather (begin(a), end(a), middle, is_a); 
cout << a << '\n'; 

/v 

9. We construct another string with underscore charactersA and some 
number values. On that input sequence, we apply gather sort. The 
gather pos iterator is the middle of the string, and the binary 
comparison function is std:: iess<char>: 

string b {"_9_2_4_7_3_8_1_6_5_0_"}; 

gather_sort (begin (b) , end(b), begin (b) + b.sizeO / 2, 
less<char>{}); 
cout << b << '\n'; 

} 

10. Compiling and running the program yields the following interesting 
output. The first three lines look like expected, but the fourth line looks 
like gather did nothing to the string. In the last line, we can see the 
result of the gather short function. The numbers appear sorted towards 
either direction: 

$ ./gather 

_aaaaaaaaaaa_ 

aaaaaaaaaaa_ 

3l3l3l3l3l3l3l3l3l3l3l 

cLd.d.d.d.d.d.d.cLd.cL 

9743201568 




How it works... 


Initially, the gather algorithm is hard to grasp because it is very short but has 

/V 

a seemingly complex task. Therefore, let'sA step through it: 

1. ) initial state 

a b c 

I 

2. ) stable_partition(a, b, [predicate); 

a b c 


3.) stable_partition(b, c, predicate); 

a b c 

* 


t 

x 




4.) return {x, y}; 


t 

y 









1. The initial state is a range of items, for which we present a predicate 

function. In the diagram, all items for which our predicate function 

returns true, are painted in gray. The iterators a and c mark the whole 

range, and iterator b points to a pivot element. The pivot element is the 

element around which we want to gather all the gray items. 

— ^ 

2. The gather algorithm calls std :: stabie partitionA on the range [a, 

/V 

b) A and while doing that, it uses a negated version of the predicate. It 

/V /V 

negates the predicateA because std:: stabie partitionA moves all 
items for which the predicate returns true to the front. We want exactly 
the opposite to happen. 

3. Another std:: stabie partition call is done but, this time, on the 
range,A [b, c) , and without negating the predicate. The gray items are 
moved to the front of the input range, which means they are all moved 

/V 

towards the pivot element pointed at byA b. 

4. The items are now gathered around b and the algorithm returns iterators 
to the beginning and the end of the now consecutive range of gray items. 

We called gather multiple times on the same range. At first, we gathered all 

the items around the middle of the range. Then we gathered the items around 

/\ 

begin () and then aroundA end () of the range. These cases are 

/\ 

interestingA because they always lead one of the 

/V 

std:: stabie partitionA calls to operate on an empty range, which results 
in no action. 

We did the last call to gather again with the parameters (begin, end, 
middle) of the range, and that did not work. Why? At first, this looks like a 
bug, but actually, it is not. 

/\ 

Imagine the character range,A "aabb", together with a predicate 

/V 

function,A is character a, which is only true for the ' a ' items—if we call it 

with a third iterator pointing to the middle of the character range, we would 

observe the same bug. The reason is that the first stabie partition call 

/\ 

would operate on the subrange,A "aa", and the other stabie partition call 

/\ _ 

operates on the range,A "bb". This series of calls cannot result in "baab", 
which we initially naively hoped. 


Note 



In order to get what we want in the last case, we could use 

std::rotate(begin, begin + 1, end); 

The gather sort modification is basically the same as gather. The only 
difference is that it does not accept a unary predicate function but a binary 
comparison function, just like std:: sort. And instead of calling 

std: : stable_partition twice, it calls std: : stable_sort twice. 


s /v 

The negation of the comparison function cannot be done withA not f n, just 

/\ 

like we did in the gather algorithmA because not fn does not work on 
binary functions. 



Removing consecutive whitespace between 
words 


Because strings are often read from user input, they may contain wild 

/V 

formatting and often need to be sanitized. One example of this isA strings 

/V 

containing too manyA whitespace. 

In this section, we will implement a slick whitespace filtering algorithm, 

/V 

which removes excess whitespace from stringsA but leaves single whitespace 
characters untouched. We call that algorithm remove muiti whitespace, and 
its interface will look very STL-like. 

How to do it... 


In this section, we will implement the remove muiti whitespace algorithm 
and check out how it works: 

/V 

1. As always, we do some includes firstA and then declare that we use the 
std namespace by default: 

#include <iostream> 

#include <string> 

#include <algorithm> 

using namespace std; 


2. We implement a new STL-style algorithm called 

remove muiti whitespace. This algorithm removes clustered 
occurrences of whitespace, but no single spaces. This means that a string 
line "a b" stays unchanged, but a string like "a a a b" is shrunk to "a 
b". In order to accomplish this, we use std::unique with a custom 

_ /v /\ 

binary predicate function. TheA std: :unqiueA walks through 

/\ 

anA iterable range and always looks at consecutive pairs of payload 
items. Then it asks the predicate functions whetherA two items are 




/V 

equal. If they are, then std:: unique removes one of them. A Afterward, 

the range does not contain subranges with equal items sitting next to 

/\ 

each other. Predicate functions thatA are usually applied in this context 

/V /V 

tell whetherA two items are equal. What we do, is giveA std: : unique a 
predicate, which tells if there are two consecutive spacesA in order to 

/V 

get those removed. Just likeA std: : unique, we accept a pair of 
begin/end iterators, and then return an iterator pointing to the new end of 
the range: 


template <typename It> 

It remove_multi_whitespace(It it. It end_it) 

{ 

return unique(it, end_it, [](const auto &a, const auto 
return isspace (a) && isspace (b); 

}) ; 

} 

3. That is already it. Let's construct a string that contains some unnecessary 
whitespace: 

int main() 

{ 

string s {"fooo bar \t baz"}; 

cout << s << 1 \n 1 ; 

4. Now, we use the erase-remove idiom on the string in order to get rid of 
the excess whitespace characters: 

s.erase(remove_multi_whitespace(begin(s), end(s)), end( 
cout << s << 1 \n 1 ; 

} 

5. Compiling and running the program yields the following output: 

$ ./remove_consecutive_whitespace 
fooo bar baz 
fooo bar baz 


How it works... 



We solved the whole complexity of the problem without any loop or manual 
comparison of items. We only provided a predicate function, which tells if 

/V _ 

twoA given characters are whitespace characters. Then we fed that predicate 
into std:: unique and poof all the excess whitespace vanished. While this 
chapter also contains some recipes where we had to fight a bit more to 
express our programs with STL algorithms, this algorithm is a really nice and 
short example. 

How does this interesting combination work in detail? Let's have a look at a 
possible implementation of std:: unique first: 

template<typename It, typename P> 

It unique(It it. It end, P p) 

{ 

if (it == end) { return end; } 

It result {it}; 
while (++it != end) { 

if (!p(*result, *it) && ++result != it) { 

*result = std::move(*it); 

} 

} 

return ++result; 

} 

The loop steps over the range items, while they do not satisfy the predicate 
condition. At the point where an item satisfies the predicate, it moves such an 
item one item past the old position, where the predicate fired the last time. 

/V 

The version of std: :uniqueA that does not accept an additional predicate 
function checks whether two neighbor items are equal. This way, it wipes out 

/V /\ 

repeated A characters as it can , for example, transformA "abbbbbbc" to 

"abc". 


What we want is not wiping out all characters which are repetitive, but 
repetitive whitespace. Therefore, our predicate does not say "both argument 
characters are equal”, but " both argument characters are whitespace 
characters 

One last thing to note is that neither std::unique nor 

remove muiti whitespace really removes character items from the 

underlying string. They only move characters within the string according to 



their semantics and tell where its new end is. The removal of all now-obsolete 
characters from the new end till the old end must still be done. This is why 
we wrote the following: 

s.erase(remove_multi_whitespace(begin(s), end(s)), end(s)); 


This adheres to the erase-remove idiom, which we already know from 
vectors and lists. 



Compressing and decompressing strings 


This section deals with a relatively popular task in coding interviews. The 
basic idea is a function, which takes a string like "aaaaabbbbbbbccc" and 

/v 

transforms it to a shorter string "a5b7c3". It is "a5"A because there are 

/V /V /V 

five A ' a ' characters. And then it is "b7 " A because there are sevenA ' b ' 
characters. This is a very simpleA compression algorithm. For normal text, it 
is of reduced utility because normal language is usually not so repetitive that 
its text representation would become shorter with this compression scheme. 
However, it is relatively easy to implement even if we have to do it on a 
whiteboard without a computer. The tricky part is that it is easy to write a 
buggy code if the program is not structured very well from the beginning. 
Dealing with strings is generally not a hard thing, but the chances 

/V /V 

ofA implementing buffer overflow bugs lurk around a lot hereA if legacy C- 
style formatting functions are used. 

Let's try an STL approach to implementing string compression and 
decompression using this simple scheme. 

How to do it... 


/V /V /V 

In this section, we willA implement simpleA compress andA decompress 
functions for strings: 

1. We include some STL libraries first, then we declare that we use 

/\ /\ 

theA stdA namespace: 

#include <iostream> 

#include <string> 

#include <algorithm> 

#include <sstream> 

#include <tuple> 


using namespace std; 




2. For our cheap compression algorithm, we try to find chunks of text 
containing ranges of the same characters, and we compress those 
individually. Whenever we start at one string position, we want to find 
the first position where it contains a different character. We use 
std:: find to find the first character in the range, which is different than 

/V 

the character at the A current position. Afterward, we return a tuple 
containing an iterator to that first different item, the character 

/V 

variableA c, which fills the range at hand, and the number of 

/\ 

occurrences that A this subrange contains: 


template <typename It> 

tuple<It, char, size_t> occurrences (It it. It end_it) 

{ 

if (it == end_it) { return {it, '?', 0}; } 

const char c {*it}; 

const auto diff (find_if(it, end_it, 

[c](char x) { return c != x; })); 

return {diff, c, distance(it, diff)}; 

} 

3. The compress algorithm continuously calls the occurrences function. 
This way, we jump from one same character group to another. The r « 
c « n line pushes the character into the output stream and then the 
number of occurrences it has in this part of the input string. The output 

/V 

is a string stream thatA automatically grows with our output. In the end, 
we return a string object from it, which contains the compressed string: 


string compress(const string &s) 

{ 

const auto end_ it (end(s) ) ; 
stringstream r; 

for (auto it (begin(s)); it != end_it;) { 

const auto [next_diff, c, n] (occurrences(it, end_i 
r << c << n; 
it = next_diff; 

} 

return r.str(); 

} 



4. The decompress method works similarly, but it is much simpler. It 
continuously tries to get a character value out of the input stream and, 

/V /\ 

then, theA following number. From those twoA values, it can construct 
a string containing the character as often as the number says. In the end, 
we again return a string from the output stream. By the way, this 
decompress function is not safe. It can be exploited easily. Can you 
guess, how? We will have a look at this problem later: 


string decompress(const string &s) 

{ 

stringstream ss{s}; 
stringstream r; 

char c; 
siz e_t n; 

while (ss >> c >> n) { r << string(n, c); } 

return r.str(); 

} 

5. In our main function, we construct a simple string with a lot of 
repetition, on which the algorithm works very well. Let's print the 
compressed version, and then the compressed and again decompressed 
version. In the end, we should get the same string as we initially 
constructed: 

int main() 

{ 

string s {"aaaaaaaaabbbbbbbbbccccccccccc"}; 

cout << compress(s) << '\n'; 

cout << decompress(compress(s)) << '\n 1 ; 

} 

6. Compiling and running the program yields the following output: 

$ ./compress 
a9b9cl1 

aaaaaaaaabbbbbbbbbccccccccccc 


How it works... 



This program basically revolves around twoA functions: compress and 

decompress. 


The decompress function is really simple because it only consists of variable 
declarations, a line of code, which actually does something, and the following 
return statement. The code line which does something is the following one: 

while (ss >> c >> n) { r << string(n, c) ; } 

/V /V /V 

It continuously reads the character,A c, and theA counter variable,A n, out of 

/V _ 

the string stream,A ss. The stringstream class hides a lot of string parsing 
magic from us at this point. While that succeeds, it constructs a 
decompressed string chunk into the string stream, from which the final result 
string can be returned back to the caller of decompress. If c = ' a ' and n = 

5, the expression string (n, c) will result in a string with the 

/V 

content,A "aaaaa". 

The compress function is more complex. We also wrote a little helper 

/V 

function for it. We called that helper functionA occurences. So, let's first 
have a glance at occurrences. The following diagram shows how it works: 

1.) occurrences(it, end); 

it end 

i ! 


a 

a 

a 







c 

c 


i 

diff 


2.) return {diff, 'b 1 , 6}; 

^ /\ /\ 

The occurences A function accepts twoA parameters: A an iterator pointing to 
the beginning of a character sequence within a range and the end iterator of 

that range. Using find if, it finds the first character that is different 

/\ 

fromA the character initially being pointed at. In the diagram, this is the 

/\ _ 

iterator,A diff. The difference between that new position and the old iterator 









position is the number of equal items (diff - it equals 6 in the diagram). 
After calculating this information, the diff iterator can be reused in order to 

execute the next search. Therefore, we pack diff, the character of the 

/\ 

subrange, andA the length of the subrange into a tuple and return it. 

With the information lined up like this, we can jump from subrange to 
subrange and push the intermediate results into the compressed target string: 

for (auto it (begin(s)); it != end_it;) { 

const auto [next_diff, c, n] (occurrences (it, end_it)); 
r « c « n; 
it = next_diff; 

} 

There’s more... 

In step 4, we mentioned that the decompress function is not safe. Indeed, it 
can easily be exploited. 

Imagine the following input string: "aooooo". Compressing it will result in 

/v /\ /\ /\ 

the substring A "ai "A because there is only one A character, A ' a '. That is 

/V 

followed by fiveA times 'O', which will result in "05". Together, this results 

/\ 

in the compressed string "ai05". Unfortunately, this compressed stringA says 
"105 times the character 'a'". This has nothing to do with our initial input 
string. Even worse, if we decompress it, we get from a six-character string to 

a 105-character string. Imagine the same with larger numbers—the user can 

/\ 

easily blow up our heap usageA because our algorithm is not prepared for 
such inputs. 

In order to prevent this, the compress function could, for example, reject 
input with numbers, or it could mask them in a special way. And the 
decompress algorithm could take another conditional, which puts an upper 

/V 

bound on the resulting string size. I am leaving this as an exercise forA you. 



Chapter 7. Strings, Stream Classes, and 
Regular Expressions 

We will cover the following recipes in this chapter: 

• Creating, concatenating, and transforming strings 

• Trimming whitespace from the beginning and end of strings 

• Getting the comfort of std:: string without the cost of constructing 

std: : string objects 
/\ 

• Reading valuesA from user input 

• Counting all words in a file 

• Formatting your output with I/O stream manipulators 

• Initializing complex objects from file input 

• Filling containers from std: : istream iterators 

• Generic printing with std::ostream iterators 

• Redirect output to files for specific code sections 

• Creating custom string classes by inheriting from std:: char traits 

• Tokenizing input with the regular expression library 

• Comfortably pretty printing numbers differently per context on the fly 

/V 

• CatchingA readable exceptions from std:: iostream errors 




Introduction 


This chapter is devoted to string handling, parsing, and printing of arbitrary 
data. For such jobs, STL provides its I/O stream library. The library basically 
consists of the following classes, which are each depicted in gray boxes: 



The arrows show the inheritance scheme of the classes. This might look very 
overwhelming at first, but we will get to use most of these classes in this 
chapter and get familiar with them class by class. When looking at those 
classes in the C++ STL documentation, we will not find them directly with 
these exact names. That is because the names in the diagram are what we see 
as application programmers, but they are really mostly just typedefs of 
classes with a basic_ class name prefix (for example, we will have an easier 


















job searching the STL documentation for basic istream rather than 

istream ). The basic * I/O stream classes are templates that can be 

_ /\ 

specialized for different character types. TheA classes in the diagram are 

/V /V /V 

specialized onA char values.A We will use these specializationsA throughout 

/V 

the book. If we prefix those class namesA with the w character, we get 
wistream, wostream, and so on—these are the specialization typedefs for 
wchar t instead of char, for example. 

At the top of the diagram, we see std:: ios base. We will basically never 

/V 

use it directly, but it is listed for completenessA because all other classes 
inherit from it. The next specialization is std:: ios which embodies the idea 

/V /\ 

of an object which maintains a stream of data, thatA can be in good A state, 
run empty of data state (EOF), or some kind of fail state. 

The first specializations we are going to actually use are std:: istream and 
std::ostream. The "i" and the "o" prefix stand for input and output. We 
have seen them in our earliest days of C++ programming in the simplest 
examples in form of the objects std:: cout and std:: cin (but also 
std:: cerr). These are instances of those classes, which are always globally 

/V 

available. We do data output via ostreamA and input via istream. 

A class which inherits from both istream and ostream is iostream. It 
combines both input and output capabilities. When we understand how all 

/\ /V 

classes from the trio consisting ofA istream, ostream and iostreamA can be 
used, we basically are ready to immediately put all following ones to use, too: 


ifstream, ofstream and fstream inherit from istream, ostream and 
/\ 

iostreamA respectively, but lift their capabilities to redirect the I/O from and 

/V 

to files from the computer'sA filesystem. 


The istr ingstream, ostringstream and iostringstream work pretty 
analogously. They help build strings in memory, and/or consuming data from 
them. 



Creating, concatenating, and transforming 
strings 


Even C++ programmers from the very old days will know about 
std:: string. While string handling is tedious and painful in C, especially 
when parsing, concatenating, copying them, and so on, std:: string is a real 
step forward regarding simplicity and safety. 

Thanks to C++11, we don't even need to copy strings when we want to 
transfer ownership to some other function or data structure anymore because 
we can move them. This way, there's not much overhead involved in most 
cases. 


The std:: string got a few new features here and there over the last few 
standard increments. What is completely new in C++17 is 
std:: string view. We will play with both a bit (but there is another recipe, 
which concentrates more on std::string view- only features) to get a 

/V 

feeling ofA them and how they work in the C++17 era. 

How to do it... 


/\ 

We willA create strings and string views and do basic concatenation and 
transformation with them in this section: 

1. As always, we first include header files and declare that we use 

/V /\ 

theA stdA namespace: 

#include <iostream> 

#include <string> 

#include <string_view> 

#include <sstream> 

#include <algorithm> 


using namespace std; 




2. Let's first create string objects. The most obvious way is instantiating an 
object a of class string. We control its content by giving the 
constructor a C-style string (which will be embedded in the binary as a 
static array containing characters after compiling). The constructor will 
copy it and make it the content of string object a. Alternatively, instead 
of initializing it from a C-style string, we can use the string literal 
operator " "s. It creates a string object on the fly. Using that to construct 
object b, we can even use automatic type deduction: 

int main() 

{ 

string a { "a" }; 
auto b ( "b"s ); 


3. The strings we just created are copying their input from the constructor 
argument into their own buffer. In order to not copy, but reference the 
underlying string, we can use string view instances. This class does 
also have a literal operator, and it is called " "sv: 

string_view c { "c" }; 
auto d ( "d"sv ); 

4. Okay, now let's play with our strings and string views. For both types, 

/V 

there areA operator« overloads for the std: :ostream class, so they 
can be printed comfortably: 

cout << a << ", " << b << 1 \n 1 ; 
cout << c << ", " << d << 1 \n 1 ; 

/v 

5. The string class overloadsA operator+, so we can add two strings and 

/V /V 

get their concatenation as a result. This way, A "a" + "b"A results in 
"ab". Concatenating a and b this way is easy. With a and c, it is not that 
easy, because c is not a string, but a string view. We have to get the 
string out of c first, and this can be done by constructing a new string 
from c, and then adding it to a. At this point one could ask, "Wait, why 
are you copying c into an intermediate string object just in order to add 
it to a? Can't you avoid that copy by using c . data () ?" That is a nice 
idea, but it has a flaw— string view instances do not have to carry zero- 




/V 

terminated strings. And this is a problem that A can lead to buffer 
overflows: 

cout << a + b << '\n 1 ; 

cout << a + string{c} << 1 \n 1 ; 

6. Let's create a new string, which contains all of the strings and string 

views we just created. By using std::ostringstream, we can print any 

/\ 

variable into a stream object thatA behaves exactly like std:: cout, but 
it doesn't print to the shell. Instead, it prints into a string buffer. After we 
streamed all the variables with some separating space between them 
using operator«, we can construct and print a new string object from 
that with o. str o : 

ostringstream o; 

o << a << " " << b << " " << c << " " << d; 
auto concatenated (o.str ()); 
cout << concatenated << '\n'; 


7. We can now also transform that new string by converting all its letters to 
upper case, for example. The C library function toupper, which maps 
lower-case characters to upper-case characters and leaves other 
characters unchanged, is already available and can be combined with 

/V 

std:: transformA because a string is basically also an iterable container 
object with char items: 


transform(begin(concatenated), end(concatenated), 
begin(concatenated), ::toupper); 

cout << concatenated << '\n'; 

} 

8. Compiling and running the program leads to the following output, which 
is just what we expected: 

$ ./creating_strings 

a, b 

c, d 

ab 

ac 



abed 
A B C D 


How it works... 

Obviously, strings can be added with the + operator like numbers, but that has 
nothing to do with math but results in concatenated strings. In order to mix 
this with string view, we need to convert to std:: string first. 

However, it is really important to note that when mixing strings and string 
views in code, we must never assume that the underlying string behind a 
string view is zero terminated\ This is why we would rather write "abc"s + 
string{some_string_view} than "abc"s + some_string_view.data() . 
Aside from that, std:: string provides a member function,A append, which 
can handle string view instances, but it alters the string instead of returning 
a new one with the string view content appended. 

Note 


std:: string view is useful, but be cautious when mixing it with 
strings and string functions. We cannot assume that they are zero- 
terminated, which breaks things quickly in a standard string 
environment. Fortunately, there are often proper function 
overloads, which can deal with them the right way. 

If we want to do complex string concatenation with formatting and so on, we 
should however not do that piece by piece on string instances. The 
std: : stringstream, std: : ostringstream, and std: :istringstream classes 
are much better suited for this, as they enhance the memory management 
while appending, and provide all the formatting features we know from 

/V /V 

streams in general. TheA std: :ostringstreamA class is what we chose in 

/v 

this sectionA because we were going to create a string instead of parsing it. 
An std:: istringstream instance could have been instantiated from an 
existing string, which we could have then comfortably parsed into variables 

of other types. If we want to combine both, std:: strings tream is the perfect 

/\ 

all-rounder.A 



Trimming whitespace from the beginning 
and end of strings 


Especially when obtaining strings from user input, they are often polluted 
with unneeded white space. In another recipe, we removed excess whitespace 
that occurred between words. 

/V 

Let's now have a look at strings thatA are surrounded by whitespace and 

/V 

remove that. TheA std:: string has some nice helper functions for getting 
this job done. 

Note 


After reading this recipe that shows how to do this with plain 
string objects, make sure to also read the following recipe. There 
we will see how to avoid unnecessary copies or data 
modifications with the new std:: string view class. 


How to do it... 


/V 

In this section, we willA write a helper function that identifies surrounding 
white space in a string and returns a copy without that, and then we are going 
to test it briefly. 

1. As always, the header includes and using directive come first: 

#include <iostream> 

#include <string> 

#include <algorithm> 

#include <cctype> 


using namespace std; 




2. Our function to trim whitespace surrounding a string takes a const 
reference to an existing string. It will return a new string without any 
surrounding whitespace: 


string trim_whitespace_surrounding(const string &s) 

{ 

3. The std: : string provides two handy functions, which help us a lot. 
The first is string: : find first_not_of , which accepts a string 
containing all the characters we want to skip over. This is, of course, 
whitespace, meaning the characters space ' ', tab ' \t ', and new 

/V 

line,A ' \n'. It returns us the first non-whitespace character position. If 
there is only whitespace in the string, it returns string : : npos. This 

means that there is only an empty string left if we trim whitespace from 

/\ 

it. So, in such aA case, let's just return an empty string: 

const char whitespace[] {" \t\n"}; 

const size_t first (s.find_first_not_of(whitespace)); 
if (string::npos == first) { return {}; } 

/v 

4. We know now where the new string has to begin, but we don'tA yet 
know where it has to end. Therefore, we use the other handy string 
function string:: f ind_iast_not_of . It will return us the last character 
position in the string which is no whitespace: 

const size_t last (s.find_last_not_of(whitespace)); 

5. Using string :: substr, we can now return the part of the string, which 
is surrounded by whitespace but without the white space. This function 
takes two parameters—a position in the string to begin with and the 

/V 

number of characters A after this position: 

return s.substr (first, (last - first + 1)); 

} 

6. That's it. Let's write a main function in which we create a string 

/V 

thatA surrounds a text sentence with all kinds of whitespace, in order to 
trim it: 


int main() 



{ 


string s {" \t\n string surrounded by ugly" 
" whitespace \t\n "}; 


7. We print the untrimmed and trimmed versions of the string. By 
surrounding the string with brackets, it's more obvious which whitespace 
belonged to it prior to trimming: 

cout << "{" << s << "}\n"; 
cout << " { " 

<< trim_whitespace_surrounding(s) 

« "}\n"; 

} 

8. Compiling and running the program yields us the output we expected: 

$ ./trim_whitespace 
{ 

string surrounded by ugly whitespace 
} 

{string surrounded by ugly whitespace} 


How it works... 

In this section, we used string:: find_first_not_of and 

string:: find iast not of . Both functions accept a C-style string, which 

acts as a list of characters that should be skipped while searching a different 

/\ /\ 

character. If we have a string instance thatA carries the string, A "too bar", 

/V 

and we call f ind f irst not of ("bf o ") on it, it will return us the valueA 5, 

/V /V 

because the 1 a 1 character is the first one that is not in theA "bfo "A string. 
The order of the characters in the argument string is not important. 

The same functions exist with inverted logic, although we did not use them in 
this recipe: string: : find_first_of and string: : find_last_of. 

Similar to iterator based functions, we need to check if these functions return 

/\ /\ 

an actual position in the stringA or a value thatA denotes that they did not 
find a character position fulfilling the constraints. If they did not find one, 



they return string: : npos. 


From the character positions we retrieved from these functions in our helper 
function, we built us a substring without surrounding whitespace, using 
string:: substring. This function accepts a relative offset and a string 
length and then returns a new string instance with its own memory, 

/V 

whichA contains only that substring. For 

/V 

example,A string! "abcdef"}. substr (2, 2) will return us a new string 
"cd". 



Getting the comfort of std::stringA without 
the cost of constructing std::string objects 


/V yv 

The std:: stringA class is a really useful classA because it simplifies 

y\ 

dealing with strings so much. A flawA is that if we want to pass around a 
substring of it, we need to pass a pointer and a length variable, two iterators, 

/V 

or a copy of the substring. We did that in the previousA recipe, where we 
removed the surrounding whitespace from a string by taking a copy of the 

y\ 

substring range that does not contain the surrounding whitespace.A 

/V 

If we want to pass a string or a substring to a library thatA does not even 
support std:: string, we can only provide a raw string pointer, which is a bit 

y\ 

disappointing, because it A sets us back to the old C days. Just as with the 
substring problem, a raw pointer does not carry information about the string 
length with it. This way, one would have to implement a bundle of a pointer 
and a string length. 

In a simplified way, this is exactly what std:: string view is. It is available 
since C++ 17 and provides a way to pair a pointer to some string together with 
that string's size. It embodies the idea of having a reference type for arrays of 
data. 

y\ yv 

If we design functions which formerly acceptedA std:: stringA instances as 
parameters, but did not change them in a way that would require the string 
instances to reallocate the memory that holds the actual string payload, we 

y\ 

could now use std:: string viewA and be more compatible with libraries 
that are STL-agnostic. We could let other libraries provide a string view 
view on the payload strings behind their complex string implementations and 
then use that in our STL code. This way, the string view class acts as a 

y\ 

minimal and useful interface, whichA can be shared among different 
libraries. 

y\ y\ 

Another cool thing is that string viewA can be used as aA non-copy 
reference to substrings of larger string objects. There are a lot of possibilities 




to use it profitably. In this section, we will play around with stri ng view in 

/\ 

order to get a feeling forA its ups and downs. We will also see how we can 
hide the surrounding whitespace from strings by adapting string views 
instead of modifying or copying the actual string. This method avoids 
unnecessary copying or data modification. 

How to do it... 


/V 

We are going to implement a function thatA relies on some string view 
features, and then, we see how many different types we can feed into it: 

1. The header includes and using directive come first: 

#include <iostream> 

#include <string_view> 

using namespace std; 


2. We implement a function that accepts a string view as its only 
argument: 

void print(string view v) 

{ 

3. Before doing anything with the input string, we remove any leading and 

/V 

trailingA whitespace. We are not going to change the string, but the view 
on the string by narrowing it down to the actual non-whitespace part of 
the string. TheA f ind f irst_not_of function will find the first 
character in the string, which is not space (' '), not a tab character 
(' \t'), and not a newline character (' \n '). With remove pref ix, we 
advance the internal string view pointer to the first non-whitespace 
character. In case the string contains only whitespace, 

/V /V /V 

theA f ind f irst_not_of A function returns the valueA npos, which is 
size_type (-1) . As size type is an unsigned variable, this boils down 
to a very large number. So, we take the smaller one of both: 
words beginA or the string view's size: 

const auto words_begin (v.find_first_not_of(" \t\n")); 
v.remove prefix(min(words begin, v.size())); 



4. We do the same with trailing whitespace. TheA remove suf f ix shrinks 
down the view's size variable: 

const auto words_end (v.find_last_not_of(" \t\n")); 
if (words_end != string_view::npos) { 

v.remove_suffix(v.size() - words_end - 1); 

} 


5. Now we can print the string view and its length: 


} 


cout << "length: " << v.length() 
« " [" « v « "]\n"; 


6. In our main function, we play around with the new print function by 
feeding it with completely different argument types. First, we give it a 

/V 

runtime char* string from the argvA pointer. At runtime, it contains the 
file name of our executable. Then, we give it an empty 
string viewA instance. We then feed it with a C-style static character 
string, and with a " "sv literal, which constructs us a string view on the 
fly. And finally, we give it an std:: string. The nice thing is that none 

/V 

of these arguments are modified or copied in order to call the A print 
function. No heap allocations happen. For many and/or large strings, this 
is very efficient: 


int main(int argc, char *argv[]) 

{ 

print(argv[0] ) ; 
print ({}); 

print("a const char * array"); 

print("an std::string_view literal"sv); 

print("an std::string instance"s); 

7. We did not test the whitespace removal feature. So, let's give it a string 

/\ 

thatA has a lot of leading and trailing whitespace: 


print (" \t\n foobar \n \t "); 



8. Another cool feature is that the strings string view gives us access to 
do not have to be zero-terminated. If we construct a string, such 

/V 

as A "abc", without a trailing zero, the print function can still safely 
handle it because string_view also carries the size of the string it points 
to: 


char cstr[] {'a', 'b', ' c'}; 

print(string_view(cstr, sizeof (cstr))); 


9. Compiling and running the program yields the following output. All the 
strings are correctly handled. The string we filled with lots of leading 
and trailing whitespace is correctly filtered, and the abc string without 
zero termination is also correctly printed without any buffer overflows: 

$ ./string view 

length: 17 [./string_view] 

length: 0 [] 

length: 20 [a const char * array] 
length: 27 [an std::string_view literal] 
length: 23 [an std::string instance] 
length: 6 [foobar] 
length: 3 [abc] 

How it works... 


We have just seen that we can call a function that accepts a string view 
argument with basically anything that is string like in the sense that it stores 
characters in a contiguous way. No copy of the underlying string was made in 
any of our print calls. 

/v 

It is interesting to note that in ourA print (argv [ o ]) call, the string view 
automatically determined the string length because this is a zero-terminated 

string by convention. The other way around, one cannot assume that it is 

/\ 

possible to determine a string viewA instances's data length by counting the 
number of items until a zero terminator is reached. Because of this, we must 
always be careful about where we reach around a pointer to the string view 
data using string view:: data (). Usual string functions mostly assume zero 



termination and, thus, can buffer overflow very badly with raw pointers to the 

/V 

payload of a string view. It is always better to use interfaces thatA already 
expect a string view. 

Apart from that, we get a lot of the luxury interface we know from 

/\ 

std: :stringA already. 

Note 

Use std:: string view for passing strings or substrings where 
you want to avoid copies or heap allocations, without losing the 
comfort of string classes. But be aware of the fact that 
std::string view drops the assumption that strings are zero 
terminated. 



Reading valuesA from user input 




/V /V /V 

A lot of recipes in this book read valuesA from anA inputA source, such 
asA standard input or a file, and doA something with it. This time we 
concentrate only on the reading and learn more about error handling, which 
becomes important if reading something from a stream did not go well and 
we need to handle it other than terminating the whole program. 

We will only read from user input in this recipe, but as soon as we know how 
to do that, we also know how to read from any other stream. User input is 
read via std:: cin, and that is essentially an input stream object, such 

/V 

asA instances of ifstream and istringstream are. 


How to do it... 

In this section, we are going to read user input into different variables, and 
see how to handle errors, as well as how to do a little bit more complex 
tokenizing of input into useful chunks: 

1. We only need iostream this time. So, let's include this single header and 

/V 

declare that we use theA std namespace by default: 

#include <iostream> 
using namespace std; 


2. Let's first prompt the user to enter two numbers. We will parse them into 
an int and a double variable. The user can separate them with 

/V 

whiteA space. 1 2 . 3 , for example, is a valid input: 


int main() 

{ 

cout << "Please Enter two numbers:\n> " 
int x; 
double y; 




3. Parsing and error checking is done at the same time in the condition part 
of our if branch. Only if both the numbers could be parsed are they 
meaningful to us and we print them: 


if (cin >> x >> y) { 

cout << "You entered: " << x 
<< " and " << y « '\n' ; 


4. If the parsing did not succeed for any reason, we tell the user that the 
parsing did not go well. The cin stream object is now in a fail state and 
will not give us other input until we clear the fail state again. In order to 
be able to parse a new input afterward, we call cin.ciearo and drop all 
input we received until now. The dropping is done with cin.ignore, 
where we specify that we are dropping the maximum number of 
characters until we finally see a newline character, which is also 
dropped. Everything after that is interesting input again: 

} else { 

cout << "Oh no, that did not go well!\n"; 
cin.clear(); 
cin.ignore( 

std::numeric_limits<std::streamsize>::max() , 

1 \n 1 ); 

} 

5. Let's now ask for some other input. We let the user enter names. As 
names can consist multiple words separated by spaces, the space 

character is not a good separator any longer. Therefore, we use 

/\ 

std: : getiine, which accepts a stream object, such asA cin, a string 
reference where it will copy the input into, and a separating character. 
Let's choose comma (,) as the separating character. By not just using 

/V 

cin alone and by usingA cin » ws as a stream parameter for 

/\ 

get line A instead, we can make cin drop any leading whitespace before 
any name. In every loop step, we print the current name, but if a name is 
empty, we drop out of the loop: 

cout << "now please enter some " 

"comma-separated names:\n> 



for (string s; getline(cin >> ws, s, { 

if (s.emptyO) { break; } 
cout << "name: \"" << s << "\"\n"; 

} 

} 

6. Compiling and running the program leads to the following output, in 
which we assumingly entered only valid inputs. The numbers are " 1 2 ", 

/V 

which areA parsed correctly, and then we enter some names which are 
then also listed correctly. An empty name input in the form of two 
consecutive commas quits the loop: 

$ ./strings_from_user_input 
Please Enter two numbers: 

>12 

You entered: 1 and 2 

now please enter some comma-separated names: 

> john doe,ellen ripley, alice,chuck norris,, 

name: "john doe" 

name: "ellen ripley" 

name: "alice" 

name: "chuck norris" 


7. When running the program again, while entering bad numbers inA the 
beginning, we see that the program correctly takes the other branch, 
drops the bad input and correctly continues with the name listening. Play 
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around with the cin. clear () and cin. ignore (...) A lines to see how 
that tampers with the name reading code: 

$ ./strings_from_user_input 
Please Enter two numbers: 

> a b 

Oh no, that did not go well! 

now please enter some comma-separated names: 

> bud spencer, terence hill,, 
name: "bud spencer" 

name: "terence hill" 


How it works 



We did some complex input retrieval in this section. The first noticeable 
thing is that we always did the retrieval and error checking at the same time. 


The result of the expression cin » x is again a reference to cin. This way, 
we can write cin » x » y » z » .... At the same time, it is possible to 
convert it into a Boolean value by using it in a Boolean context such 
asA if A conditions. A The Boolean value tells us if the last read was 
successful. That is why we were able to write if (cin » x » y) {...}. 

If we, for example, try to read an integer, but the input contains "foobar" as 
the next token, then parsing this into the integer is not possible and the stream 
object enters a fail state. This is only critical for the parsing attempt but not 
for the whole program. It is okay to reset it and then to try anything else. In 
our recipe program, we tried to read a list of namesA after a potentially 

failing attempt to read two numbers. In the case of a failing attempt to read 

/\ 

those numbers in, we usedA cin. clear () to put cin back into a working 
state. But then, its internal cursor was still on what we typed instead of 
numbers. In order to drop this old input and clear the pipe for the names 
input, we used the very long 

/V 

expression, A cin.ignore(std::numeric_limits<std::streamsize>::max() 

' \n');. This is necessary to clear whatever is in the buffer at this point, 
because we want to start with a really fresh buffer when we ask the user for a 
list of names. 

The following loop might look strange at first, too:A 

for (string s; getline (cin >> ws, s, { ... } 

In the conditional part of the for loop, we use getline. 

TheA getiineA function accepts an input stream object, a string reference as 
an output parameter, and a delimiter character. By default, the delimiter 
character is the newline symbol. Here, we defined it to be the comma (,) 

/V 

character, so all the names in a list, such asA " john, carl, frank", are read 
individually. 

/V 

So far, so good. But what does it mean to provide theA cin » ws function 

_ /V 

as a stream object? This makes A cin first flush all the whitespace, which lead 
before the next non-whitespace character and after the last comma. Looking 



/V 

back at theA " john, carl, frank" example, we would get the substrings 
" john", " carl", and " frank" without using ws. Notice the unnecessary 
leading space characters for carl and frank? These effectively vanish 
because of our ws pretreatment of the input stream. 



Counting all words in a file 


Let's say we read a text file and we want to count the number of words in the 
text. We define that one word is a range of characters between whitespace 
characters.A How do we do it? 

We could count the number of spaces, for example, because there must be 
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spaces between words. In the sentence,A "John has a funny little 
dog. ", we have five space characters, so we could say there are six words. 

/V 

What if we have a sentence with whitespace noise, such asA " A a John 
a a has a a \t a a\nfunny little dog ."? There are way too many 
unnecessary spaces in this string, and it's not even only spaces. From the 
other recipes in this book, we already learned how we can remove such 
excess whitespace. So, we could first preprocess the string into a normal 

/V 

sentence formA and then apply the strategy of counting space characters. 

Yes, that is doable, but there is a much easier way. Why shouldn't we use 
what the STL already provides us? 

In addition to finding an elegant solution for this problem, we will let the user 
choose if we shall count the words from the standard input or a text file. 

How to do it... 


In this section, we will write a one-liner function that counts the words from 
an input buffer, and let the user choose where the input buffer reads from: 

1. Let's include all the necessary headers first and declare that we use 
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theA stdA namespace: 

#include <iostream> 

#include <fstream> 

#include <string> 

#include <algorithm> 

#include <iterator> 




using namespace std; 


/v 

2. Our wordcount function accepts an input stream, for example, A cin. It 
creates an std:: input iteratorA iterator, which tokenizes the strings 
out of the stream and then feeds them to std:: distance. 

. /v /v 

TheA distanceA parameter accepts two iterators as arguments and tries 
to determine how many incrementing steps are needed in order to get 
from one iterator position to the other. For random access iterators, this 

/V 

is simpleA because they implement the mathematical difference 
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operation (operator-). Such iteratorsA can be subtracted from each 
other like pointers. An istreamiterator however, is a forward 

/v 

iteratorA and must be advanced until it equals the end iterator. 
Eventually, the number of steps needed is the number of words: 

template ctypename T> 
size_t wordcount(T &is) 

{ 

return distance(istream_iterator<string>{is}, {}); 

} 

3. In our main function, we let the user choose if the input stream 

/V /V /\ 

willA beA std:: cinA or an input file: 

int main(int argc, char **argv) 

{ 

size t wc; 


4. If the user launches the program in the shell together with a file name 

/V 

(such asA $A . / count_all_words some_textf ile . txt), then we obtain 
that filename from the argv command-line parameter array and open it, 
in order to feed the new input file stream into wordcount: 

if (argc == 2) { 

ifstream ifs {argv[l]}; 
wc = wordcount(ifs); 

5. If the user launched the program without any parameter, we assume that 
the input comes from standard input: 



} else { 

wc = wordcount(cin); 

} 

6. That's already it, so we just print the number of words we saved in the 
variable wc! 


cout << "There are " << wc << " words\n"; 

}; 

7. Let's compile and run the program. First, we feed the program from 
standard input without any file parameter. We can either pipe an echo 
call with some words into it or launch the program and enter some 
words from the keyboard. In the latter case, we can stop the input by 
pressing Ctrl+D. This is how echoing some words into the program 
looks: 

$ echo "foo bar baz" | ./count_all_words 

There are 3 words 


8. When launching the program with its source code file as input, it will 
count how many words it consists of: 

$ ./count_all_words count_all_words.cpp 
There are 61 words 


How it works... 

There is not much left to say; most of it has been explained while 
implementing it as this program is very short. One thing we could elaborate 
on a bit is the fact that we used std:: cin and an std:: if stream instance in a 

/V /V /V 

completely interchangeable way. TheA cin is of theA std:: istreamA type, 
and std:: if stream inherits from std:: istream. Have a look at the class 
inheritance diagram at the beginning of this chapter. This way, they are 
completely interchangeable, even at runtime. 


Note 



Keep your code modular by using stream abstractions. This helps 
decouple source code parts and makes your code easy to test 
because you can just inject any other matching type of stream. 



Formatting your output with I/O stream 
manipulators 


In many cases, just printing out strings and numbers is not enough. 
Sometimes, numbers need to be printed as decimal numbers, sometimes as 
hexadecimal, and sometimes even as octal. Sometimes we want to see a "Ox" 
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prefix in front of hexadecimal numbers, sometimes not. A 

When printing floating-point numbers, there are also a lot of things we may 
want to have an influence on. Should the decimal values always be printed 
with the same precision? Should they be printed at all? Or perhaps,A we want 
a scientific notation? 

Apart from scientific presentation and hexadecimal, octal, and so on, we also 
want to present the user output in a tidy form. Some output can be arranged 
in tables, for example, in order to make it as readable as possible. 

All these things are, of course, possible with output streams. Some of these 
settings are also important when parsing values from input streams. In this 
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recipe, we will get a feeling ofA such so-called I/O manipulators by playing 

/V 

around with them. Sometimes, they appear tricky, soA we will also get 

/V 

into A some details. 

How to do it... 

In this section, we will print numbers with wildly varying format 
settings,A in order to get familiar with I/O manipulators: 

1. First, we include all the necessary headers and declare that we use 
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the A stdA namespace by default: 


#include <iostream> 
#include <iomanip> 
#include <locale> 




using namespace std; 


/v 

2. Next, we define a helper function thatA prints a single integer value with 
different styles. It accepts a padding width and a filling character for 
padding, which is set to space ' ' by default: 


void print_aligned_demo(int val, 

size_t width, 

char fill_char = ' ') 

{ 

3. With setw, we can set the minimum number of characters output for 

/\ 

printing a number. If we print 123 with a width ofA 6, for example, we 
get" a 12 3", or "12 3 a ". We can control on which side the padding 
OCCUrs with std: : lett, std: : right, and std : : internal. When 
printing numbers in the decimal form, internal looks identical to 
right. But if we print the value Oxi, for example, with a width of 6 and 

/V _ /V 

withA internal, we get "Ox a 6". The setfiiiA manipulator defines 
the character thatA will be used for padding. We will try different styles: 

cout << "================\n"; 

cout << setfill(fill_char); 

cout << left << setw(width) << val << '\n 1 ; 
cout << right << setw(width) << val << '\n'; 
cout << internal << setw(width) << val << '\n 1 ; 

} 


4. In the main function, we start using the function we just implemented. 

/V 

At first, we print the value A 12345 , with a width of 15 . We do this 
twice, but the second time, we use the ' _' character for padding: 


int main() 

{ 

print_aligned_demo(123456, 15); 
print_aligned_demo(123456, 15, '_'); 

5. Afterward, we print the value 0xi2 3abc with the same width as before. 

/V /V /V 

However,A before doing this, we applyA std: :hexA and 

std:: showbase to tell the output stream object cout that it should print 





/\ 

numbers in the hexadecimal formatA and that it should prepend "Ox" to 
them so that it is obvious that they are to be interpreted as hex: 


cout << hex << showbase; 
print_aligned_demo(0xl23abc, 15); 

6. We can do the same with oct, which tells cout to use the octal system 

_ /V /V /V 

for printing numbers. TheA showbase is still active, soA oA will be 
prepended to every printed number: 

cout << oct; 

print_aligned_demo(0123456, 15); 


7. With hex and uppercase, we get the ' x ' in "Ox" printed upper case. The 
' abc ' in ' 0xi23abc ' is also upper cased: 

cout << "A hex number with upper case letters: " 

<< hex << uppercase << 0xl23abc << '\n'; 

/v 

8. If we want to printA 100 in the decimal format again, we have to 
remember that we switched the stream to hex before. By using dec, we 
can put it back to normal: 

cout << "A number: " << 100 << '\n'; 

cout << dec; 

cout << "Oops, now in decimal again: " << 100 << '\n'; 


9. We can also configure how Boolean values are printed. By default, true 

/V 

is printed as lA and false as o. With booiaipha, we can set it to a text 
representation: 

cout << "true/false values: " 

<< true << ", " << false << '\n'; 
cout << booiaipha 

<< "true/false values: " 

<< true << ", " << false << '\n'; 

/v 

10. Let's have a look at floating-point variables of theA float and 
/\ /\ 

double A types. If we print a number such as A 12 . 3 , it is printed as 



/V 

12 .3, of course. If we have a number such asA 12 . 0 , the output stream 
will just drop the decimal point, which we can change with showpoint. 
Using this, the decimal point is always displayed: 

cout << "doubles: " 

« 12.3 « ", " 

« 12.0 « ", " 

<< showpoint << 12.0 << '\n'; 

11. The representation of a floating-point number can be scientific or fixed, 
scientific means that the number is normalized to such a form that the 

/V 

first digit is the only digitA before the decimal point, and then the 
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exponent is printed, which is needed to multiply the numberA backA to 
its actual size. For example, the value 300 . o would be printed as 
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"3. 0 E 2 ", because 300 A equalsA 3.0 * io A 2 . fixed reverts back to the 
normal decimal point notation: 

cout << "scientific double: " << scientific 
« 123000000000.123 « '\n'; 

cout << "fixed double: " << fixed 

« 123000000000.123 « '\n'; 

12. Apart from the notation, we can also decide with what precision a 
floating-point number is printed. Let's create a very small value and 
print it with 10 digits after the decimal point, and once with just one 
digit after the decimal point: 

cout << "Very precise double: " 

<< setprecision (10) << 0.0000000001 << 1 \n 1 ; 

cout << "Less precise double: " 

<< setprecision (1) << 0.0000000001 << 1 \n 1 ; 

} 


13. Compiling and running the program yields us the following lengthy 
output. Those fourA first blocks of output are from the print helper 
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function that A tampered around with the A setw and 
left/right/internal modifiers. Afterward, we played with the casing 
of base representations, Boolean representation, and floating-point 
formatting. It is a good idea to play with each of these to get familiar 



with them: 


$ ./formatting 


123456 

123456 

123456 


123456 


123456 

123456 


0xl23abc 

0xl23abc 

Ox 123abc 


0123456 

0123456 

0123456 

A hex number with upper case letters: 0X123ABC 
A number: 0X64 

Oops, now in decimal again: 100 
true/false values: 1, 0 
true/false values: true, false 
doubles: 12.3, 12, 12.0000 
scientific double: 1.230000E+11 
fixeddouble: 123000000000.123001 
Very precise double: 0.0000000001 
Less precise double: 0.0 


How it works... 


/V 

All these, sometimes pretty long,A « foo « bar stream expressions are 
really confusing if it is not clear to the reader what each of them does. 
Therefore, let's have a look at a table of existing formatting modifiers. They 

are all to be placed in a input_stream >> modifier or output stream << 
modifier expression and then affect the following input or output: 


Symbol 


Meaning 











setprecision (int Sets the precision parameter when printing or parsing 
n) floating-point values. 


showpoint / 

/v 

A noshowpoint 


Enables or disables the printing of the decimal point of 
floating-point numbers even if they do not have any 
decimal places. 


fixed / 
scientific / 

/v 

A hexfloat / 

/v 

A defaultfloaf 


Numbers can be printed in a fixed style (which is the 
most intuitive one) or scientific style, fixed and 
scientific stand for these modes, hexfloat activates 

both modes, which formats floating-point numbers in 
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hexadecimalA floating-point notation.A defauitfloat 
deactivates both modes. 


showpos / 
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A noshowpos 


Enable or disable printing a ' +' prefix for positive 
floating-point values. 


Read or write exactly n characters. When reading, this 
setw (int n) truncates the input. When printing, padding is applied if 

/V 

the output would be A shorter than n characters. 


When applying padding (see setw), fill the output 

/\ /\ _ 

withA character values,A c. The default is space (' '). 


internalA/left 

/v 

/A right 


left and right control where the padding for fixed- 

/V 

width prints (see setw) occurs. internalA puts padding 
characters in the middle between integers and their 
negative sign, the hex prefix and a hexadecimally printed 

/V 

value,A or monetary units and values. 


Integral values can be printed and parsed in the decimal, 



dec / hex / oct 

hexadecimal, and octal base systems. 

setbase(int n) 

This is the numeric synonymous function to dec/hex/oct, 
which are equivalent if used with the values 10 / 16 / 8 . 
Other values reset the base choice to 0 , which leads to 
decimal printing again, or parsing based on the prefix of 
the input. 

quoted(string) 

Prints string in quotes or parse from quoted input, and 
then drops the quotes, string can be a String class 
instance or a C-style character array. 

boolalpha / 

/\ 

A noboolalpha 

Prints or parses Boolean values as/from alphabetical 
representation rather than 1/0 strings. 

showbase / 

/V 

A noshowbase 

Enables or disables base-prefixes when printing or 
parsing numbers. For hex, this is Ox; for octal it is 0 . 

uppercase / 
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A nouppercase 

Enables or disables upper casing or alphabetical 
characters when printingA floating-point and 
hexadecimal values. 

The best way to get familiar with those is studying their variety a bit and 
playing with them. 

When playing with them, however, we might have noticed already that 
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mostA of these modifiers appear to be sticky A and a few of them, not so. 

Sticky means that once applied, they appear to influence the input/output 

/\ 

forever A until they are reset again. The only non-sticky ones from this table 
are setw and quoted. They only affect the next item in the input/output. This 

/V 

is important to knowA because if we print some output with certain 



formatting, we should tidy up our stream object formatting settings afterward, 
because the next output from unrelated code may otherwise look crazy. Same 
applies to input parsing, where things can break with the wrong I/O 
manipulator options. 

/\ 

We did not really use any of thoseA because they do not have to do anything 
with formatting, but for the reason of completeness, we should also have a 
look at some other stream state manipulators: 

Symbol Meaning 

/V 

skipws / Enables or disables the feature ofA input streams skipping 

noskipws whitespace 

unitbuf / Enables or disables immediate output buffer flushing after 

nouni tbuf any output operation 

ws Can be used on input streams to skip any whitespace at the 

head of the stream 

ends Writes a string-terminating ' \ o ' character into a stream 

flush Immediately flushes out whatever is in the output buffer 

endl Inserts a ' \ n' character into an output stream and flushes the 

output 

From these, only skipws/noskipws and unitbuf/nounitbuf appear sticky. 



Initializing complex objects from file input 


Reading in individual integers, floats, and word strings is really easy, because 
the » operator of input stream objects is overloaded for all these types, and 
input streams conveniently drop all in-between whitespace for us. 

But what if we have a more complex structure that we want to read from an 
input stream, and if we need to read strings that contain more than one word 
(as they would normally be chunked into single words because of the 
whitespace skipping)? 

For any type, it is possible to provide another input stream operator» 
overload, and we are going to see how to do it.A 

How to do it... 


/V 

In this section, we'llA define a custom data structure and provide facilities to 
read such items from input streams as standard input: 

1. We need to include some headers first and for comfort, we declare that 
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we use theA std namespace by default: 

#include <iostream> 

#include <iomanip> 

#include <string> 

#include <algorithm> 

#include <iterator> 

#include <vector> 

using namespace std; 


2. Asa complex object example, we define a city structure. A city shall 
have a name, a population count, and geographic coordinates: 



struct city { 

string name; 
size_t population; 
double latitude; 
double longitude; 

}; 


3. In order to be able to read such a city from a serial input stream, we need 
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to overload the stream functionA operator». In this operator, we first 
skip all the leading whitespace with ws, because we do not want 
whitespace to pollute the city name. Then, we read a whole line of text 
input. This implies that in the input file, there is a whole text line only 
carrying the name of a city object. Then, after a newline character, a 
whitespace-separated list of numbers follows, indicating the population, 
the geographic latitude, and the longitude: 


istreamS operator»(istream &is, city &c) 
{ 

is >> ws; 

getline(is, c.name); 
is >> c.population 
>> c.latitude 
>> c.longitude; 
return is; 


/v 

4. In our main function, we create a vector that A can hold a range of city 
items. We fill it using std:: copy. The input of the copy call is an 
istream iterator range. By giving it the city struct type as a template 
parameter, it will use the operator» function overload, which we just 
implemented: 


int main() 

{ 

vector<city> 1; 

copy(istream_iterator<city>{cin}, {}, 

back inserter(1)); 


5. In order to see whether our city parsing went right, we print what we got 



in the list. The I/O formatting,A left « setw ( 15 ) «, leads to the city 
name being filled with whitespace, so we get our output in a nicely 
readable form: 


for (const auto &[name, pop, lat. Ion] : 1) { 

cout << left << setw(15) << name 
<< " population=" << pop 
<< " lat=" << lat 
<< " lon=" << Ion << '\n'; 

} 

} 

6. The text file from which we will feed our program looks like this. There 

/V 

are fourA example cities with their population count and geographical 
coordinates: 


Braunschweig 

250000 52.268874 10.526770 
Berlin 

4000000 52.520007 13.404954 
New York City 

8406000 40.712784 -74.005941 
Mexico City 

8851000 19.432608 -99.133208 

7. Compiling and running the program yields the following output, which 
is what we expected. Try to tamper around with the input file by adding 
some unnecessary whitespace before the city names in order to see how 
it gets filtered out: 

$ cat cities.txt| ./initialize_complex_objects 

Braunschweigpopulation=250000 lat=52.2689 lon=10.5268 
Berlinpopulation=4000000 lat=52.52 lon=13.405 
New York City population=8406000 lat=40.7128 lon=-74.0059 
Mexico City population=8851000 lat=19.4326 lon=-99.1332 


How it works... 


This was another short recipe again. The only thing we did was creating a 

/V 

new struct city, then we overloaded std:: istreamA iterator's 
operator»A for this type and that's it. This already enabled us to deserialize 



city items from standard input using istream_iterator<city>. 

There might be an open question left regarding error checking. For that, let's 
have a look at the operator» implementation again: 

istream& operator>>(istream &is, city &c) 

{ 

is >> ws; 

getline(is, c.name); 

is >> c.population >> c.latitude >> c.longitude; 
return is; 

} 

We are reading a lot of different things. What happens if one of them fails 
and the next one doesn't? Does that mean that we are potentially reading all 
following items with a bad "offset" in the token stream? No, this cannot 
happen. As soon as one of these items cannot be parsed from the input 
stream, the input stream object enters an error state and refuses to parse 
anything further. This means that if for example c. population or 
c. latitude cannot be parsed, the remaining » operands just "drop through", 
and we leave this operator function scope with a half-deserialized city object. 

/\ 

On the caller side, we are notified by this when we writeA i f 

(input stream » city object ). Such a streaming expression is implicitly 

A 

converted to a bool value when used as aA conditional expression. It returns 

/\ 

faiseA if the input stream object is in an error state. Knowing that we can 
reset the stream and do whatever is appropriate. 

In this recipe, we did not write such i f conditionals ourselves because we 
/\ ... 

letA std: : istream_iterator<city> do the deserialization. The operator++ 

implementation of this iterator class also checks for errors while parsing. If 
any errors occur, it will refuse iterating further. In this state, it returns true 
when it is compared to the end iterator, which makes the copy algorithm 
terminate. This way, we are safe. 



Filling containers from std::istream 
iterators 


In the last recipe, we learned how we can assemble compound data structures 
from an input stream and then fill lists or vectors with those. 

This time, we make it a little bit harder by filling an std: :map from standard 
input. The problem here is that we cannot just fill a single structure with 
values and push it back into a linear container like a list or a vector is because 
map divides its payload into key and value parts. It is, however, not 
completely different, as we will see. 

After studying this recipe, we will feel comfortable with serializing and 
deserializing complex data structures from and to character streams. 

How to do it... 


We are going to define another structure like in the last recipe, but this time 

/V 

we are going to fill it into a map, which makes it more complicatedA because 
this container maps from keys to values instead of just holding all values in a 
list: 

1. First, we include all the needed headers and declare that we use 

/V 

theA std namespace by default: 

#include <iostream> 

#include <iomanip> 

#include <map> 

#include <iterator> 

#include <algorithm> 

#include <numeric> 

using namespace std; 


2. We want to maintain a little Internet meme database. Let's say a meme 




/V 

has a name, a description, and theA year when it was born or invented. 
We will save them in an std: :map, where the name is the key, and the 
other information is bunched up in a struct as the value associated with 
the key: 


struct meme { 

string description; 
size_t year; 

}; 


/v 

3. Let's first ignore the key and just implement a streamA operator» 
function overload for struct meme. We assume that the description is 
surrounded by quotation marks, followed by the year. This would look 
like "some description" 2017 in a text file. As the description is 
surrounded by quotation marks, it can contain whitespace because we 

know that everything between the quotation marks belongs to it. By 

/\ 

reading with A is » quoted(m. description), the quotation marks are 
automatically used as delimiters and dropped afterward. This is very 
convenient. Just after that, we read the year number: 


istream& operator>>(istream &is, meme &m) { 

return is >> quoted(m.description) >> m.year; 

} 

4. OK, now we take the meme's name as the key for the map into account. 
In order to insert a meme into the map, we need an 
std: :pair<key_type, value_type> instance. key_type is string, of 
course, and vaiue type is meme. The name is allowed to contain spaces 
too, so we use the same quoted wrapper as for the description, p. first 
is the name and p. second is the whole meme structure associated with it. 
It will be fed into the other operator» implementation that we just 
implemented: 


istreamS operator >>(istream &is, 

pair<string, meme> &p) { 

return is >> quoted(p.first) >> p.second; 

} 



5. Okay, that's it. Let's write a main function, which instantiates a map, and 

/V 

fill that map. Because we overloadedA the stream function operator», 
istream iterator can deal with this type directly. We let it deserialize 
our meme items from standard input and use an inserter iterator in 
order to pump them into the map: 

int main() 

{ 

map<string, meme> m; 

copy(istream_iterator<pair<string, meme»{cin}, 

{ } , 

inserter(m, end(m))); 


6. Before we print what we have, let's first find out what's the longest 
meme name in the map. We use std:: accumulate for this. It gets an 

/V 

ini tial value Ou (u for unsigned)A and will visit the map element-wise in 
order to merge them together. In terms of accumulate, merging usually 
means adding. In our case, we want no numeric sum of anything, but the 

/V 

largest A string length. In order to get that, we provide accumulate a 
/\ 

helper function, A max func, which takes the current maximum size 

/V 

variable (which must be unsignedA because string lengths are unsigned) 
and compares it to the length of the current item's meme name string, in 
order to take the maximum of both values. This will happen for each 

/V /V 

element. TheA accumula teA function's final return value is the 
maximum meme name length: 


auto max_func ([](size_t old_max, 

const auto &b) { 

return max(old_max, b.first.length()); 

}) ; 

size_t width {accumulate(begin(m), end(m), 

Ou, max_func)}; 

7. Now, let's quickly loop through the map and print each item. We use « 

/V 

left « setw (width) A to get a nice table-like printing: 

for (const auto &[meme_name, meme_desc] : m) { 
const auto &[desc, year] = meme desc; 



cout << left << setw(width) << meme_name 
<< " : " << desc 
<< ", " << year << '\n 1 ; 

} 

} 

8. That's it. We need a small Internet meme database file, so let's fill a text 
file with some examples: 

"Doge" "Very Shiba Inu. so dog. much funny, wow." 2013 
"Pepe" "Anthropomorphic frog" 2016 
"Gabe" "Musical dog on maximum borkdrive" 2016 
"Honey Badger" "Crazy nastyass honey badger" 2011 
"Dramatic Chipmunk" "Chipmunk with a very dramatic look" 20 


9. Compiling and running the program with the example meme database 
yields the following output: 

$ cat memes.txt | ./filling_containers 

Doge: Very Shiba Inu. so dog. much funny, wow., 2013 

Dramatic Chipmunk : Chipmunk with a very dramatic look, 200 

Gabe: Musical dog on maximum borkdrive, 2016 

Honey Badger: Crazy nastyass honey badger, 2011 

Pepe: Anthropomorphic frog, 2016 


How it works... 


There were three specialties in this recipe. One was that we did not fill a 
normal vector or a list from a serial character stream, but a more complex 
container like std: :map. The other was that we used those magic quoted 
stream manipulators. And the last was the accumulate call, which finds out 

/V 

the largest A key string size. 

Let's start with the map part. Our struct meme only contains a description 

/v _ 

field andA year. The name of the Internet meme is not part of this 

/V 

structure A because it is used as the key for the map. When we insert 
something into a map, we can provide an std: :pair with a key type and a 
value type. This is what we did. We first implemented stream operator» for 



struct meme, and then we did the same for pair<string, meme>. Then we 

/V /V 

USedA istream_iterator<pair<string, meme»{cin}A to get Such items 

/\ 

out of the standard input, and fed them into the map using A inserter (m, 
end(m)). 


When we deserialized meme items from the stream, we allowed the names 
and descriptions to contain whitespace. This was easily possible, although we 

/V 

only used one line per memeA because we quoted those fields. An example 
of the line format is as follows; "Name with spaces" "Description with 
spaces" 123 

/\ 

When dealing with quoted stringsA both in input and output, std:: quoted is 

/V 

a great help. If we have a string,A s, printing it using cout « quoted (s) 
will put it in quotes. If we deserialize a string from a stream, for example, 

/V 

viaA cin » quoted (s) , it will read the next quotation mark, fill the string 
with what is following, and continue until it sees the next quotation mark, no 

/\ /V 

matter how manyA whitespace areA involved. 

The last strange looking thing was max func in our accumulate call: 

auto max_func ([](size_t old_max, const auto &b) { 

return max(old_max, b.first.length()); 

}) ; 

size_t width {accumulate(begin(m), end(m), Ou, max_func)}; 

/v 

Apparently, max func accepts a size t argumentA and another auto-typed 
argument which turns out to be a pair item from the map. This looks really 
weird at first as most binary reduction functions accept arguments of identical 
types and then merge them together with some operation, just as std: :pius 
does. In this case, it is really different because we are not merging actual 
pair items. We only pick the key string length from every pair, drop the rest, 
and then reduce the resulting size_t values with the max function. 

In the accumulate call, the first call of max func gets the Ou value we initially 

/\ 

provided as the left argumentA and a reference to the first pair item on the 
right side. This results in a max (Ou, string iength) return value, which is 
the left argument in the next call with the next pair item as the right 
parameter, and so on. 



Generic printing with std::ostream iterators 


It is pretty easy to print anything with output streams, as the STL is already 

/\ /\ 

shipped with many useful operator«A overloads for the most basicA types. 
This way, data structures containing items of such types can easily be printed 

/V 

using the std: :ostream_iterator class, which we've already doneA quite 

/v 

often inA this book. 

In this recipe, we will concentrate on how to do this with a custom type, and 
what else we can do to manipulate printing via template type choices without 
much code at the caller side. 


How to do it... 


We will play with std:: ostream iterator by enabling for combination with 
a new custom class and have a look into its implicit conversion capabilities, 
which can help us with printing: 

1. The include files come first and then we declare that we use the std 
namespace by default: 

#include <iostream> 

#include <vector> 

#include <iterator> 

#include <unordered_map> 

#include <algorithm> 

using namespace std; 


2. Let's implement a transformation function, which maps numbers to 
strings. It shall return "one" for the value 1 , "two" for the value 2 , and 
so on: 


string word_num(int i) { 


3. We fill a hash map with the mappings we need in order to access them 




later: 


unordered_map<int, string> m { 

{1, "one"}, {2, "two"}, {3, "three"}, 

{4, "four"}, {5, "five"}, //... 

}; 


/v 

4. Now, we can feed the hash map's find function with the argument,A i, 
and return what it finds. If it doesn't find anything, because there is no 

/V 

translation for a given number, we return the string,A " unknown": 

const auto match (m.find(i)); 

if (match == end(m)) { return "unknown"; } 

return match->second; 


5. Another thing with which we will play later with is struct bork. It only 

/\ 

contains an integerA and is also implicitly constructible from an integer. 
It has a print function, which accepts an output stream reference and 

/V 

prints theA "bork" string repeatedly, depending on the value of its 
member integer borks: 

struct bork { 
int borks; 

bork(int i) : borks{i} {} 


void print(ostreamS os) const { 

fill_n(ostream_iterator<string>{os, 
borks, "bork!"s); 

} 


}, 


}; 


/V 

6. In order to gain convenience with borkmprintA we overload 

operator« for stream objects, so they automatically call bork::print 
whenever bork objects are streamed into an output stream: 


ostreams operator« (ostream &os, const bork &b) { 

b.print(os); 
return os; 

} 


7. Now we can finally begin implementing the actual main function. We 



initially just create a vector with some example values: 


int main() 

{ 

const vector<int> v {1, 2, 3, 4, 5}; 

8. Objects of type ostream iterator need a template parameter, which 
denotes which type of variables they can print. If we write 

ostream_iterator<T>, it will later use ostreams operator (ostreams, 

/\ _ 

const ts) A for printing. This is exactly what we implemented before 

/V 

for theA bork type, for example. This time, we are just printing integers, 
so it is ostream iterator<int>. It shall use cout for printing, so we 
provide it as the constructor parameter. We go through our vector in a 

/V 

loopA and assign each item i to the dereferenced output iterator. This is 

/V 

how streamA iterators are used by STL algorithms too: 

ostream_iterator<int> oit {cout}; 

for (int i : v) { *oit = i; } 
cout << '\n'; 


9. The output of the iterator we just produced is fine, but it prints the 

/V 

number without any separator. If we wantA a bit of separating 
whitespace between all printed items, we can provide a custom spacing 

/V 

string as aA second parameter of the output stream iterator's constructor. 
This way, it prints " l, 2 , 3, 4, 5, " instead of "12345". 
Unfortunately, we cannot easily tell it to drop the comma-space string 

after the last number, because the iterator does not know of its 

/\ 

endA before it reaches it: 

ostream_iterator<int> oit_comma {cout, ", 

for (int i : v) { *oit_comma = i; } 
cout << 1 \n 1 ; 


10. Assigning items to an output stream iterator in order to print them is not 
a wrong way to use it, but this is not what they were invented for. The 
idea is to use them in combination with algorithms. The simplest one is 



std :: copy. We can provide the begin and end iterators of the vector as 
an input range and the output stream iterator as the output iterator. It will 
print all the numbers of the vector. Let's do that with both the output 
iterators and later compare the output with the loops we wrote before: 

copy(begin(v), end(v), oit); 
cout << 1 \n 1 ; 


copy(begin(v), end(v), oit_comma); 
cout << 1 \n 1 ; 


/v 

11. Remember the function,A word num, which maps numbers to strings, as 
l to "one", 2 to "two", and so on? Yes, we can use those for printing 

too. We just need to use an output stream operator, which is template 

/\ 

specialized on stringA because we are not printing integers any longer. 
And instead of std:: copy, we use std:: transform because it allows us 
to apply a transformation function to each item in the input range before 
copying it to the output range: 


transform(begin(v) , end(v), 

ostream_iterator<string>{cout. 


word_num); 

cout << 1 \n 1 ; 


VI TV 




/v 

12. The last output line in this program finally puts struct borkA to 

/V 

use.A We could, but do not provide a transformation function to 

/\ 

std:: transform. A Instead, we can just create an output stream iterator, 

/V 

which is specialized on theA bork type in an std:: copy call. This leads 

/\ 

to theA bork instances being implicitly created from the input range 
integers. That will give us some interesting output: 


} 


copy(begin(v), end(v) , 

ostream iterator<bork>{cout, "\n"}); 


13. Compiling and running the program yields us the following output. The 

/V /\ 

first twoA lines are completely identical to the next twoA lines, which is 
what we suspected. Then, we get nice, written-out number strings in a 
line, followed by a lot of bork ! strings. These occur in multiple 



/V 

linesA because we used a "\n" separator string instead of spaces for 
those: 

$ ./ostream_printing 
12345 

1 , 2 , 3 , 4 , 5 , 

12345 

1 , 2 , 3 , 4 , 5 , 

one two three four five 
bork! 

bork! bork! 
bork! bork! bork! 
bork! bork! bork! bork! 
bork! bork! bork! bork! bork! 


How it works... 


We have seen that std:: ostream iterator is really just a syntax hack, 
which kind of squeezes the act of printing into the form and syntax of an 

/V 

iterator. Incrementing such an iterator does nothing. DereferencingA it only 
returns us a proxy object whose assignment operator forwards its argument to 
an output stream. 

Output stream iterators that are specialized on a type t (as in 
ostream iterator<T>) WOl'k with all types for which an ostreamS 
operator« (ostreams, const t&) implementation is provided. 

/V 

ostream iteratorA always tries to call operator« for the type it was 
specialized for, via its template parameter. It will try to implicitly convert 
types if the same is allowed. When we iterate over a range of A-typed items 
but we copy those items over to output_iterator<B> instances, this will 
work if a is implicitly convertible to b. We did exactly the same thing with 

/V 

struct bork: aA bork instance is implicitly convertible from an integer 
value. That is why it was so easy to throw a lot of "bork !" strings onto the 
user shell. 

If implicit conversion is not possible, we can do that ourselves, using 
std:: transform, which is what we did in combination with the word num 
function. 




Note 


/\ 

Note that it is, in general,A bad style to allow implicit 
conversions for custom types because this is a common source of 
bugs that are really hard to find later. In our example use case, the 

implicit constructor is more useful than dangerous because the 

/\ 

class is used for nothing else butA printing. 



Redirecting output to files for specific code 
sections 


The std:: cout provides a really nice way to print whatever we want, 

/V 

whenever we wantA because it is simple to use, easily extensible, and 

/\ 

globally accessible. Even if we want to print special messages, such as A error 
messages, which we want to isolate from normal messages, we can just use 

/V 

std:: cerr, which is the same as cout A but prints to the standard error 
channel instead of the standard output channel. 

We might have some more complicated desires for logging sometimes. Let's 
say, for example, we want to redirect the output of a function to a file, or we 
want to mute the output of a function, without changing the function at all. 

/V 

Perhaps,A it is a library function we cannot access the source code of. 

Maybe, it was never designed to write to a file but we want its output in a 
file. 

It is indeed possible to redirect the output of stream objects. In this recipe, we 
are going to see how to do that in a very simple and elegant way. 

How to do it... 

/V 

We are going to implement a helper class thatA solves the problem of 
redirecting a stream and reverting that redirection again with 
constructor/destructor magic. And then we see how we can put it to use: 

1. We only need the headers for input, output, and file streams this time. 

/V 

And we declare theA std namespace as a default namespace for lookup: 

#include <iostream> 

#include <fstream> 


using namespace std; 




2. We implement a class, which holds a file stream object and a pointer to 

_ /\ 

a stream buffer. TheA cout as a stream object has an internal stream 

buffer, which we can simply exchange. And while we exchange it, we 

/\ 

can save what it was before, so we canA undo any change later. We 
could look its type up in the C++ reference, but we can also use 
decitype to find out what type cout. rdbuf () returns. This is not 
generally good practice in all situations, but in this case, it's just a 
pointer type: 

class redirect_cout_region 
{ 

using buftype = decitype(cout.rdbuf()); 

ofstream ofs; 
buftype buf_backup; 


3. The constructor of our class accepts a filename string as its only 

parameter. The filename is used to initialize theA file stream 

/\ 

member,A ofs. After initializing it, we can feed it into cout as a new 

_ /V 

stream buffer. The same function thatA accepts the new buffer also 

/\ 

returns a pointer to the old one, so we can save it in order toA restore it 
later: 


public: 

explicit 

redirect_cout_region (const string Sfilename) 

: ofs{filename}, 

buf_backup{cout.rdbuf(ofs.rdbuf())} 

{ } 

4. The default constructor does the same as the other constructor. The 
difference is, that it does not open any file. Feeding a default- 
constructed file stream buffer into the cout stream buffer leads to cout 
being kind of deactivated. It will just drop its input we give it for 
printing. This can also be useful in some situations: 

redirect_cout_region () 

: ofs{}, 

buf backup{cout.rdbuf(ofs.rdbuf())} 




5. The destructor just restores our change. When an object of this class 
runs out of scope, the stream buffer of cout is the old one again: 

~redirect_cout_region() { 

cout.rdbuf(buf_backup); 

} 


6. Let's mock an output-heavy function, so we can play with it later: 

void my_output_heavy_function() 

{ 

cout << "some output\n"; 

cout << "this function does really heavy work\n"; 
cout << "... and lots of it...\n"; 

// . . . 

} 


7. In the main function, we first produce some completely normal output: 


int main() 

{ 

cout << "Readable from normal stdout\n"; 

8. Now we're opening another scope, and the first thing we do in this scope 
is instantiating our new class with a text file parameter. File streams 
open files in read and write mode by default, so it creates this file for us. 
Any following output will now be redirected to this file, although we use 
cout for printing: 

{ 

redirect_cout_region _ {"output.txt"}; 
cout << "Only visible in output.txt\n"; 
my_output heavy function (); 

} 

9. After leaving the scope, the file is closed and the output is redirected to 
the normal standard output again. Let's now open another scope in 
which we instantiate the same class, but via its default constructor. This 



way the following printed line of text will not be visible anywhere. It 
will just be dropped: 


{ 


} 


redirect_cout_region 

cout << "This output will " 

"completely vanish\n"; 


10. After leaving that scope also, our standard output is resurrected and the 
last line of text output will be readable in the shell again: 

cout << "Readable from normal stdout again\n"; 

} 

11. Compiling and running the program yields the output as we expected it. 
Only the very first and the very last lines of output are visible in the 
shell: 

$ ./log_regions 

Readable from normal stdout 

Readable from normal stdout again 


/\ /v 

12. We can see that a new file,A output. txt, has beenA created and 
contains the output of the first scope. The output of the second scope 
vanishes completely: 

$ cat output.txt 

Only visible in output.txt 

some output 

this function does really heavy work 
... and lots of it... 


How it works... 

Every stream object has an internal buffer for which it acts as a front end. 

_ /V 

Such buffers are exchangeable. If we have a stream object, A s, and want to 

/V /V 

save its buffer into a variable,A a, and install a new buffer,A b, this looks like 

/V 

the following:A a = s . rdbuf (b) . Restoring it can be simply done with 



s.rdbuf(a). 


This is exactly what we did in this recipe. Another cool thing is that we can 
Stack those redirect_cout_region helpers: 

{ 

cout << "print to standard output\n"; 

redirect_cout_region la {"a.txt"}; 
cout << "print to a.txt\n"; 

redirect_cout_region lb {"b.txt"}; 
cout << "print to b.txt\n"; 

} 

cout << "print to standard output again\n"; 

This works because objects are destracted in the opposite order of their 
construction. The concept behind this pattern that uses the tight coupling 

/V 

between construction and destruction of objects is calledA Resource 
Acquisition Is Initialization (RAII). 

There is one really important thing that should be mentioned—the 
initialization order of the member variables of 

/V /V 

theA redirect_cout_regionA class: 

class redirect_cout_region { 

using buftype = decltype(cout.rdbuf ()); 

ofstream ofs; 
buftype buf_backup; 

public: 

explicit 

redirect_cout_region(const string Sfilename) 

: ofs{filename}, 

buf_backup{cout.rdbuf(ofs.rdbuf())} 

{ } 


/v 

As we can see, the member,A bufjoackup, is constructed from an expression 

/V 

that depends on ofs. A This obviously means that ofs needs to be initialized 
before bufbackup. Interestingly, the order in which these members are 



/V 

initialized does not depend on the order of theA initializer list items. The 
initialization order only depends on the order of the member declarations ! 

Note 

If one class member variable needs to be initialized after another 
member variable, they must also appear in that order in the class 
member declaration. The order of their appearance in the 
initializer list of the constructor is not critical. 



Creating custom string classes by inheriting 
from std::char traits 


/v 

The std:: string is extremely useful. However,A as soon as people need a 
string class with slightly different semantics for string handling, some tend to 
write their own string class. 

/V 

Writing your own string class is rarely a good ideaA because safe 

/\ 

stringA handling is hard. Fortunately, std:: string is only a specializing 

/V 

typedef of the template class,A std: :basic_string. This class contains all 
the complicated memory handling stuff, but it does not impose any policy on 
how strings are copied, compared, and so on. This is something thatA is 
imported into basic string by accepting a template parameter that contains 
a traits class. 

In this recipe, we will see how to build our own trait classes and, this way, 

/\ 

how to A create custom strings without reimplementing anything. 

How to do it... 

We are going to implement two different custom string classes: lc string 
and ci string. The first class constructs lower case strings from any string 
input. The other class does not transform any string, but it can do case- 
insensitive string comparison: 

1. Let's include the few necessary headers first and then declare that we use 

/V /V 

the A stdA namespace by default: 

#include <iostream> 

#include <algorithm> 

#include <string> 


using namespace std; 





^ /V /\ 

2. Then we reimplement theA std:: toiowerA function, which is already 
defined in <cctype>. The already existing function is fine, but it is not 

/V 

constexpr. Some string functions are constexprA since C++ 17, 
however, and we want to be able to make use of that with our own 
custom string trait class. The function maps upper-case characters to 
lower caseA and leaves other characters unchanged: 


static constexpr char tolow(char c) { 
switch (c) { 

case 'return c - 'A' + 'a'; 
default: return c; 

} 

} 


3. The std::basic_string class accepts threeA template parameters: the 

underlying character type, a character traits class, and an allocator type. 

/\ 

We are only changing the character traits class in this sectionA because 
it defines the behavior of strings. In order to reimplement only what 
should differ from the ordinary strings, we are publicly inheriting from 
the standard traits class: 


class lc_traits : public char_traits<char> { 
public: 


/v 

4. Our class accepts input stringsA but transforms them to lower case. 

There is one function, which does this character-wise, so we can put our 

/\ _ 

ownA tolow function here. This function is constexpr, which is why 
we reimplemented ourselves a constexprtoiow function: 


static constexpr 

void assign(char_type& r, const char_type& a ) { 

r = tolow (a); 

} 


, /v 

5. The other function handles the copying of an entireA string into its own 

memory. We use an std:: transform call to copy all the characters from 
the source string to the internal destination string and, at the same time, 
map every character to its lower-case version: 



}; 


static char_type* copy(char_type* dest, 

const char_type* src, 
size_t count) { 

transform(src, src + count, dest, tolow); 
return dest; 


} 


_ /\ /y 

6. The other A trait helps build a string class that A effectively transforms 

/V 

strings to lower case. We are going to write another trait thatA leaves the 
actual string payload untouched but which is case insensitive when it 
comes to comparing strings. We inherit from the existing standard 
character traits class again, and this time, we redefine some other 
member functions: 


class ci_traits : public char_traits<char> { 
public: 


/v 

7. The eq function tells whether A two characters are equal. We do this too, 
but we compare their lower-case versions. This way ' a ' equals ' a': 

static constexpr bool eq(char_type a, char_type b) { 
return tolow(a) == tolow(b); 

} 

_ /v /v 

8. The it function tells whetherA the value ofA a is less than the value of 
b. We apply the correct logical operator for that, just after lower-casing 
both the characters again: 


static constexpr bool lt(char_type a, char_type b) { 
return tolow(a) < tolow(b); 

} 

_ /v 

9. The last twoA functions worked on character-wise input and the next 
two functions work on string-wise input. The compare function works 
similar to the old-school strncmp function. It returns o if both the strings 

/V 

are equal within the length thatA count defines. If they differ, it returns 
a negative or positive number, which tells which input string is 
lexicographically smaller. Calculating the difference between both the 



characters at every position must, of course, be done on their lower-case 

/v 

versions. The nice thing is that this whole loop code has A been part of 

/\ /V 

aA constexprA function since C++14: 

static constexpr int compare(const char_type* si, 

const char_type* s2, 
size_t count) { 

for (; count; ++sl, ++s2, --count) { 

const char_type diff (tolow(*sl) - tolow(*s2)); 
if (diff < 0) { return -1; } 

else if (diff > 0) { return +1; } 

} 

return 0; 

} 

10. The last function we need to implement for our case-insensitive string 

/V /V 

class is find. For a given input string,A p, and length,A count, it finds 

/\ _ 

the position of a character, A ch. Then, it returns a pointer to the first 

/V 

occurrence of that character, or it returnsA nuiiptr if there is none. The 

/V 

comparison in this function has to be done using the toiowA "glasses" in 
order to make the search case-insensitive. Unfortunately, we cannot use 
std:: find if, because it is not constexpr, and must write a loop 
ourselves: 


static constexpr 

const char_type* find(const char_type* p, 

size_t count, 
const char_type& ch) { 
const char_type find_c {tolow(ch)}; 

for (; count != 0; --count, ++p) { 

if (find_c == tolow(*p)) { return p; } 

} 

return nuiiptr; 

} 

}; 

11. Okay, that's it for the traits. Since we have them in place now, we can 

/\ 

define twoA new string class types, lc string means lower-case string. 
ci string means case-insensitive string. Both the classes only differ 



from std:: string by their character traits class: 


using lc_string = basic_string<char, lc_traits>; 
using ci_string = basic_string<char, ci_traits>; 


/v 

12. In order to make the output streams acceptA these new classes for 
printing, we quickly need to overload the stream operator«: 


ostreamS operator« (ostreamS os, const lc_string& str) { 
return os.write(str.data(), str.size()); 

} 

ostreamS operator« (ostreamS os, const ci_string& str) { 
return os.write(str.data(), str.size()); 

} 

13. Now we can finally begin implementing the actual program. 

/\ 

Let'sA instantiate a normal string, a lower-case string, and a case- 
insensitive string, and print them immediately. They should all look 
normal on the terminal, but the lower case strings should be all lower¬ 
cased: 


int main() 

{ 

cout << " string: " 

<< string{"Foo Bar Baz"} << '\n' 

<< "lc_string: " 

<< lc_string{"Foo Bar Baz"} << '\n' 
<< "ci^string: " 

<< ci string!"Foo Bar Baz"} << '\n' 


14. In order to test the case-insensitive string, we can instantiate two strings 

/\ 

that A are basically equal but differ in the casing of some characters. 
When doing a really case-insensitive comparison, they should appear 
equal nevertheless: 


ci_string user_input {"MaGiC PaSsWoRd!"}; 
ci_string password {"magic password!"}; 

15. So, let's compare them and print that they matchA if they do: 



} 


if (user_input == password) { 

cout << "Passwords match: \"" << user_input 
<< "\" == \"" << password << "\"\n"; 

} 


16. Compiling and running the program yields us the expected results. 
When we first printed the same string three times in different types, we 

got unchanged results, but the lest ring instance is all lower case. The 

/\ 

comparison of the two strings thatA only differ in their character casing 
was indeed successful and yields us the right output: 

$ ./custom_string 
string: Foo Bar Baz 

lc_string: foo bar baz 
ci^string: Foo Bar Baz 

Passwords match: "MaGiC PaSsWoRd!" == "magic password!" 


How it works... 


All the subclassing, and function reimplementing we did will surely look a bit 
crazy for beginners. Where did all the function signatures come from, of 
which we magically knew that we need to reimplement? 

/V /V 

Let's first have a look whereA std:: stringA really comes from: 

template < 

class CharT, 

class Traits = std::char_traits<CharT>, 
class Allocator = std::allocator<CharT> 

> 

class basic_string; 

The std: : string is really an std: : basic_string<char>A and that expands 

to std::basic_string<char, std::char_traits<char>, 

std:: aiiocator<char». Okay, that is a long type description, but what does 

_ /\ 

it mean? The point of all of this is that it is possible to baseA a string not only 

/\ _ 

on single-byte char items A but also on other, larger, types. This enables for 
string types, which can handle more than the typical American ASCII 
character set. This is not something we will have a look into now. 



The char_traits<char> class, however, contains algorithms 

/V _ /V 

thatA basic string needs for its Operation. TheA char_traits<char> 
knows how to compare, find, and copy characters and strings. 

The aiiocator<char>A class is also a traits class, but its special job is 
handling string allocation and deallocation. This is not important for us at this 
time as the default behavior satisfies our needs. 

If we want a string class to behave differently, we can try to reuse as much as 
possible from what basic string and char traits already provide. And 
this is what we did. We implemented two chartraits subclasses called 

case insentitive and lower caser and configured two completely new 

/\ 

string typesA with them by using them as substitutes for the standard 

char_traits type. 


Note 

In order to explore what other possibilities there are to adapt 
basic string to your own needs, look up the C++ STL 
documentation for std:: char traits and see what other 
functions it has that can be reimplemented. 



Tokenizing input with the regular 
expression library 


When parsing or transforming strings in complex ways or breaking them into 
chunks, regular expressions are a great help. In many programming 
languages, they are already built in because they are so useful and handy. 

If you do not know regular expressions yet, have a look at 

/V /V 

the A Wikipedia A article about them, for example. They will surely extend 

/\ 

your horizon, as it is easy to see how useful they areA when parsing any kind 

/V 

of text. Regular expressions can, for example, test whetherA an e-mail 

/\ 

address string or an IP addressA string is valid, find and extract substrings 
out of large strings, which follow a complex pattern, and so on. 

In this recipe, we will extract all the l ink s out of an HTML file and list them 

_ /\ 

for the user. The code will be amazingly shortA because we have regular 
expression support built in the C++ STL since C++11. 

How to do it... 


We are going to define a regular expression that detects links, and we apply it 
to an HTML file in order to pretty print all the links that A occur in that file: 

1. Let's first include all the necessary headers, and declare that we use 

/V 

theA std namespace by default: 

#include <iostream> 

#include <iterator> 

#include <regex> 

#include <algorithm> 

#include <iomanip> 

using namespace std; 


2. We will later generate an iterable range, which consists of strings. These 




strings always occur in pairs of a link and a link description. Therefore, 
let's write a little helper function, which pretty prints these: 

template <typename Inputlt> 

void print (InputIt it, Inputlt end_it) 

{ 

while (it != end_it) { 

3. In each loop step, we increment the iterator twice and take copies of the 

l ink and the link description they contain. Between the two iterator 

/\ /\ 

dereferences, we add another guarding i f A branch thatA checks 

/\ 

whetherA we prematurely reached the end of the iterable range, just for 
safety: 


const string link {*it++}; 
if (it == end_it) { break; } 
const string desc {*it++}; 


4. Now, let's print the link with its description in a nicely prettified form 
and that's it: 


} 


cout << left << setw(28) << desc 
<< " : " << link << '\n' ; 


/v 

5. In the main function, we are reading in everything thatA comes from 
standard input. To do this, we are constructing a string from the whole 
standard input via an input stream iterator. In order to prevent 
tokenizing, because we want the whole user input as-is, we use 
noskipws. This modifier deactivates whitespace skipping and 
tokenizing: 


int main() 

{ 

cin >> noskipws; 

const std::string in {istream_iterator<char>{cin}, {}}; 

/v 

6. Now we need to define a regular expression thatA describes how we 
assume an HTML link to look. The parentheses,A (), within the regular 



expression define groups. These are the parts of the link we want to 
access—the URL it links to, and its description: 


const regex link_re { 

"<a href=\"(I A \"]*)\"[ A <]*>([ A <]*)</a>"}; 

/v •'v 

7. TheA sregextoken i ter a tor A class has the same look and feel as 
ofA istream iterator. We give it the whole string as iterable input 

range and the regular expression we just defined. There is also a third 

/\ 

parameter, A { 1 , 2 }, which is an initializer list of integer values. It 
defines that we want to iterate over the groups 1 and 2 from the 
expressions it captures: 


sregex_token_iterator it { 

begin(in), end(in), link_re, {1, 2}}; 

/v 

8. Now we have an iterator thatA will emit the links and link descriptions 
if it finds any. We provide it together with a default constructed iterator 
of the same type to the print function we implemented before: 

print (it, {}); 


9. Compiling and running the program gives us the following output. I ran 

/V ' 

theA curl program on the ISO C++ homepage, which simply 
downloads an HTML page from the Internet. Of course, it would also be 
possible to write cat some_html_file.html | ./link_extraction. 

The regular expression we used is pretty much hardcoded to a fixed 

assumption of how links look in the HTML document. It may be 

/\ 

exercised by youA to make it more general: 

$ curl -s "https://isocpp.org/blog" | ./link_extraction 

Sign In / Suggest an Article : https://isocpp.org/member/lo 

Register : https://isocpp.org/member/register 

Get Started! : https://isocpp.org/get-started 

Tour : https://isocpp.org/tour 

C++ Super-FAQ: https://isocpp.org/faq 

Blog : https://isocpp.org/blog 

Forums : https://isocpp.org/forums 



Standardization: https://isocpp.org/std 
About: https://isocpp.org/about 

Current ISO C++ status : https://isocpp.org/std/status 
(...and many more...) 


How it works... 

Regular expressions (or regex in short) are extremely useful. They can look 
really cryptic, but it is worth learning how they work. A short regex can spare 
us writing many lines of code if we did the matching manually. 

In this recipe, we first instantiated an object of type regex. We fed its 
constructor with a string that describes a regular expression. A very simple 
regular expression is ".which matches every character because a dot is the 

/\ /v 

regex wildcard. A If we write "a", then this matches only on theA ' a ' 
characters. If we write "ab*", then this means "one a, and zero or arbitrarily 
many b characters". And so on. Regular expressions are another large topic, 
and there are great explanations on Wikipedia and other websites or 
literature. 

/V 

Let's have another look at our regular expression thatA matches what we 

assume to be HTML links. A simple HTML link can look like <a 

href="some_url. com/foo">A great link</a>. We want the 

some uri. com/foo part, as well as a great link. So we came up with the 

following regular expression, which contains groups for matching substrings: 


Reg ular expression: 


Matched strin g: 



Group 1 


Group 2 


Group 0 


The whole match itself is always Group 0 . In this case, this is the full <a 

_ /V 

href . </a> string. The quoted href -part thatA contains the URL being 

linked to is Group 1 . The ( ) parentheses in the regular expression define 
such a group of which we have 2. The other one is the part between the <a 






... > and </a>, which contains the link description. 


There are various STL functions that A accept regex objects, but we directly 
used a regex token iterator adapter, which is a high-level abstraction that uses 

/V 

std : : regex searchA under the hood in order to automate recurring 
matching work. We instantiated it like this: 

sregex_token_iterator it {begin (in), end(in), link_re, {1, 2}}; 

The begin and end part denote our input string over which the regex token 
iterator shall iterate and match all links. TheA link re is, of course, the 
complex regular expression we implemented to match links. The { 1 , 2 } part 
is the next complicated looking thing. It instructs the token iterator to stop on 
each full match and first yield Group 1, then after incrementing the iterator to 
yield Group 2, and after incrementing it again, it would finally search for the 
next match in the string. This somewhat intelligent behavior really spares us 
some code lines. 

Let's have a look at another example to make sure we got the idea. Let's 

/\ 

imagine the regular expression,A "a (b*) (c*) ". It will match strings that 

/V 

contain anA a character, then none or arbitrarily many b characters, and then 
none or arbitrarily many c characters: 

const string s {" abc abbccc "}; 
const regex re {"a(b*) (c*)"}; 

sregex_token_iterator it {begin(s), end(s), re, {1, 2}}; 

print ( *it ); // prints b 
++it; 

print ( *it ); // prints c 
++it; 

print ( *it ); // prints bb 
++it; 

print ( *it ); // prints ccc 

There is also theA std: : regex iterator class, which emits the substrings 
that are between regex matches. 



Comfortably pretty printing numbers 
differently per context on the fly 


In the last recipe, we learned how to format the output with output streams. 
And while doing the same, we realized two facts: 

• Most I/O manipulators are sticky, so we have to revert their effect after 
use in order to not tamper with other unrelated code, which also prints 

• It can be very tedious and does not look very readable if we have to set 

/\ 

up long chains of I/O manipulators in order to get only aA few variables 
printed with specific formatting 

A lot of people do not like I/O streams for such reasons, and even in C++, 
they still use printf for formatting their strings. 

In this recipe, we will see how to format types on the fly without too much 
I/O manipulator noise in our code. 

How to do it... 

/\ 

We are going to implement a class,A f ormat guard, which can automatically 
revert any format setting. Additionally, we add a wrapper type, which can 
contain any value, but when it is printed, it gets special formatting without 
burdening us with I/O manipulator noise: 

1. First, we include some headers and declare that we use 

/\ /\ 

theA stdA namespace: 

#include <iostream> 

#include <iomanip> 

using namespace std; 

2. The helper class that A tidies up our stream formatting states for us is 




called f ormat guard. Its constructor saves the formatting flags, which 
std:: cout has set at the moment. Its destructor restores them to the state 

it had when the constructor was called. This effectively revokes any 

/\ 

formatting settings that were appliedA in between: 

class format_guard { 

decltype(cout.flags()) f { cout. flags ()}; 
public: 

~format_guard() { cout.flags(f); } 

}; 

3. Another little helper class is scientific type. Because it's a class 
template, it can wrap any payload type as a member variable. It basically 
does nothing: 


template ctypename T> 
struct scientific_type { 

T value; 

explicit scientific_type(T val) : value{val} {} 

}; 


4. We can define completely custom formatting settings for any type 

/V 

thatA was wrapped into scientifictype before because if we 
overload the stream operator» for it, the stream library executes 
completely different code when printing such types. This way, we can 

/V 

print scientific valuesA in scientific floating-point notation, with 

/V 

uppercase formatting and explicit +A prefix if they have positive values. 
We do also use our f ormat guard class in order to tidy up all our 
settings when leaving this function again: 


template <typename T> 

ostreamS operator«(ostream &os, const scientific_type<T> & 
format_guard 

os << scientific << uppercase << showpos; 
return os << w.value; 

} 

5. In the main function, we will first play around with the format guard 



class. We open a new scope, first get an instance of the class, and then 
we apply some wild formatting flags to std:: cout: 


int main() 

{ 

{ 

format_guard 

cout << hex << scientific << showbase << uppercase; 

cout << "Numbers with special formatting:\n"; 
cout << 0xl23abc << '\n 1 ; 
cout << 0.123456789 << '\n'; 

} 

6. After we printed some numbers with many formatting flags enabled, we 
left the scope again. While this happened, the destructor of 
format_guard tidied the formatting up. In order to test this, we are 
printing exactly the same numbers again. They should appear different: 

cout << "Same numbers, but normal formatting again:\n"; 
cout << 0xl23abc << '\n'; 
cout << 0.123456789 << '\n'; 


/v 

7. Now we putA scientific type to use. Let's print three floating-point 
numbers in a row. We wrap the second number in scientif ic type. 
This way, it is printed in our special scientific style, but the numbers 
before and after it get default formatting. At the same time, we avoid 
ugly formatting line noise: 

cout << "Mixed formatting: " 

« 123.0 « " " 

<< scientific_type{123.0} << " " 

« 123.456 « '\n'; 

} 

8. Compiling and running the program yields us the following result. The 
first two numbers are printed with specific formatting. The next two 
numbers appear with default formatting, which shows us that our 
format_guard works just nicely. The three numbers in the last lines also 
look just as expected. Only the one in the middle has the formatting of 



scientific type, the rest has default formatting: 

$ ./pretty_print_on_the_fly 
Numbers with special formatting: 

0X123ABC 
1.234568E-01 

Same numbers, but normal formatting again: 
1194684 
0.123457 

Mixed formatting: 123 +1.230000E+02 123.456 



Catching readable exceptions from 
std::iostream errors 


In none of the recipes in this chapter, we used exceptions to catch errors. 
While this is certainly possible, working on stream objects without 
exceptions is already very convenient. If we try to parse in 10 values, but this 
fails somewhere in the middle, the whole stream object sets itself into a fail 
state and stops further parsing. This way, we do not run into the danger of 
parsing variables from the wrong offset in the stream. We can just do the 

/V 

parsing in a conditional, such asA if (cin » foo » bar » ...). If this 
fails, we handle it. It does not appear very advantageous to embrace parsing 
in a try { ... } catch ... block. 

In fact, the C++ I/O stream library already existed before there were 
exceptions in C++. Exception support was added later, which might be an 
explanation why they are not a first-class supported feature in the stream 
library. 

In order to use exceptions in the stream library, we must configure each 
stream object individually to throw an exception, whenever it sets itself into a 
fail state. Unfortunately, the error explanations in the exception objects, 
which we can then catch later, are not thoroughly standardized. This leads to 
not really helpful error messages, as we will see in this section. If we really 
want to use exceptions with stream objects, we can additionally poll the C 
library for filesystem error states to get some additional information. 

In this section, we are going to write a program that can fail in different ways, 
handle those with exceptions, and see how to squeeze more information out 
of those afterward. 


How to do it... 

/V /V 

We will A implement a program that A opens a file (which might fail), and 




then we'll read an integer out of it (which might fail, too). We do this with 
activated exceptions and then we see how we can handle those: 


1. First, we include some headers and declare that we use 

/V /V 

theA stdA namespace: 

#include <iostream> 

#include <fstream> 

#include <system_error> 

#include <cstring> 

using namespace std; 


2. If we want to use stream objects with exceptions, we have to enable 
them first. In order to get a file stream object to throw an exception if the 
file we are letting it access does not exist, or if there are parsing errors, 
we need to set some fail bits in an exception mask. If we do something 
afterward that fails, it will trigger an exception. By activating faiibit 
and badbit, we enable exceptions for filesystem errors and parsing 
errors: 


int main() 

{ 

ifstream f; 

f.exceptions(f.faiibit | f.badbit); 


3. Now we can open a try block and access a file. If opening the file is 
successful, we try to read an integer from it. Only if both steps succeed, 
we print the integer: 

try { 

f.open("non_existant.txt"); 

int i; 
f >> i; 

cout << "integer has value: " << i << '\n'; 


4. In both the expected possibilities of an error, an instance of 



std : : ios base:: failure is thrown. This object has a what o A member 

function, which ought to explain what triggered the exception. 

/\ 

Unfortunately, the standardization of this messageA was left out, and it 
does not give too much information. However, we can at least 
distinguish if there is a filesystem problem (because the file does not 

exist, for example) or a format parsing problem. The global 

/\ _ 

variable,A errno, has been there even before C++ was invented, and it 
is set to an error value, which we can check now. TheA strerror 
function translates from an error number to a human readable string. If 
errno is o, there is, at least, no filesystem error: 

catch (ios_base::failures e) { 
cerr << "Caught error: 
if (errno) { 

cerr << strerror (errno) << '\n 1 ; 

} else { 

cerr << e.what() << '\n 1 ; 

} 

} 

} 

5. Compiling the program and running it in two different scenarios yields 
the following output. If the file to be opened does exist but parsing an 

/V 

integer from it was not possible, we get an iostream categoryA error 
message: 

$ ./readable_error_msg 

Caught error: ios_base::clear: unspecified iostream_categor 

/v /v 

6. If the file does A not exist, we will be notified about this A with a 
different message from strerror (errno) : 

$ ./readable_error_msg 

Caught error: No such file or directory 


How it works... 


/V 

We have seen that we can enable exceptions per stream object,A s, 

/\ 

withA s . exceptions (s. faiibit | s .badbit) . This means, that there is no 

/V /V 

way to use, for example, theA std:: ifstreamA instance's constructor for 



/V 

opening a file if we want to get an exceptionA when opening that file is not 
possible: 


ifstream f {"non_existant.txt"}; 
f.exceptions (...); // too late for an exception 


This is a pity because exceptions actually promise that they make error 
handling less clumsy compared to old-school C-style code, which is riddled 
with loads of if branches, which handle errors after every step. 

If we played around trying to provoke various reasons for streams to fail, we 
would realize that there are no different exceptions being thrown. This way, 

/V /V 

we can only find out when A we get an error, butA not what specific error 

_ /V 

(This is, of course,A not true for exception handling in general, but for the 
STL stream library). That is why we additionally consulted the value of 
errno. This global variable is an ancient construct, which has already been 
used in the old days when there were no C++ or exceptions in general. 

/V 

If any system-related function has seen an error condition, it isA able to set 
the errno variable to something other than o (o describes the absence of 

/V 

errors), and then the caller isA able to read that error number and look up 

what its value means. The only problem with this is that when we have a 

/\ 

multithreaded application, and all the threads use functions thatA can set this 

/V 

error variable, whose error value is it? If we read it even though there isA no 

error, it could carry an error value because some other system function 

/\ 

running in a different threadA may have experienced an error. Luckily, this 

/\ 

flaw has beenA gone since C++11, where every thread in a process sees its 
own errno variable. 


Without elaborating the ups and downs of an ancient error indication method, 
it can give us useful extra information when an exception is triggered on 

/V 

system-based things such asA file streams. Exceptions tell us when it 
happened, and errno can tell us what happened if it happened at the system 
level. 



Chapter 8. Utility Classes 


In this chapter, we will cover the following recipes: 

• Converting between different time units using std:: ratio 

• Converting between absolute and relative times with std: : chrono 

• Safely signalizing failure with std::optional 

• Applying functions on tuples 

• Quickly composing data structures with std:: tuple 

• Replacing void* with std::any for more type safety 

• Storing different types with std: : variant 

• Automatically handling resources with std : : unique ptr 

• Automatically handling shared heap memory with std: : shared ptr 

/V /V 

• Dealing withA weak pointers to A shared objects 

• Simplifying resource handling of legacy APIs with smart pointers 

/V 

• Sharing different member values A of the same object 

• Generating random numbers and choosing the right random number 
engine 

A _ 

• GeneratingA random numbers and letting the STL shape specific 
distributions 




Introduction 


This chapter is dedicated to utility classes that are very useful for solving 
very specific tasks. Some of them are indeed so useful that we will most 
probably see them extremely often in any C++ program snippet in the future 
or have at least already seen them sprinkled over all other chapters in this 
book. 

, /v 

The first two recipes are about measuring and taking theA time. We will also 

see how to convert between different time units and how to jump between 
points in time. 

Then, we will have a look at the optional, variant, and any types (which all 
came with C++14 and C++17) as well as some tuple tricks in another 

/V 

five A recipes. 

Since C++11, we also got sophisticated smart pointer types, namely 

/V /V 

unique ptr, shared_ptr,A and weak ptr, whichA are an enormously 
effective help regarding memory management , which is why we will have a 
dedicated look at them in five recipes. 

Finally, we will have a panoramic view ofA the library parts of the STL that 
are about generating random numbers. Apart from learning about the most 
important characteristics of the STL's random engines, we will also leam how 
to apply shaping to random numbers in order to get distributions that fit our 
actual needs. 




Converting between different time units 
using std:: ratio 


Since C++11, the STL contains some new types and functions for taking, 
measuring, and displaying time. This part of the library exists in the 
std:: chrono namespace and has some sophisticated details. 

In this recipe, we will concentrate on measuring time spans and how to 
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convert the result of the measurement between units, such asA seconds, 
milliseconds, and microseconds. The STL provides facilities, which enable us 
to define our own time units and convert between them seamlessly. 

How to do it... 


In this section, we will write a little game that prompts the user to enter a 
specific word. The time that the user needs to type this word into the 
keyboard is measured and displayed in multiple time units: 

1. At first, we need to include all the necessary headers. For reasons of 
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comfort, we declare that we use theA std namespace by default: 

#include <iostream> 

#include <chrono> 

#include <ratio> 

#include <cmath> 

#include <iomanip> 

#include <optional> 

using namespace std; 


/v 

2. The chrono:: durationA as a type for time durations usually refers to 
multiples or fractions of seconds. All the STL time duration units refer 
to integer typed duration specializations. In this recipe, we are going to 
specialize on double. In the recipe after this one, we will concentrate 

/V /\ 

more on the existing time unitA definitionsA that are already built into 




the STL: 


using seconds = chrono::duration<double>; 

3. One millisecond is a fraction of a second, so we define this unit by 
referring to seconds. The ratio multiply template parameter applies 
the STL-predefined milli factor to seconds : : period, which gives US 

_ /v /v 

the fraction we want. TheA ratio_muitipiyA template is basically a 
meta programming function for multiplying ratios: 

using milliseconds = chrono::duration< 

double, ratio_multiply<seconds::period, milli»; 


4. It's the same thing with microseconds. While a millisecond is a mi in¬ 
fraction of a second, a microsecond is a micro-fraction of a second: 

using microseconds = chrono::duration< 

double, ratio_multiply<seconds::period, micro»; 

5. Now we are going to implement a function, which reads a string from 
user input and measures how long it took the user to type the input. It 
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takes no argumentsA and returns us the user input string as well as the 
elapsed time, bundled in a pair: 

static pair<string, seconds> get_input() 

{ 

string s; 

6. We need to take the time from the beginning of the period during which 
user input occurs and after it. Taking a time snapshot looks like this: 

const auto tic (chrono::steady_clock::now()); 


7. The actual capturing of user input takes place now. If we are not 

successful, we just return a default-initialized tuple. The caller will see 
that he got an empty input string: 

if ( ! (cin >> s)) { 

return {{}, {}}; 


} 



8. In the case of success, we continue by taking another time snapshot. 
Then we return the input string and the difference between both time 
points. Note that both are absolute time points, but by calculating the 
difference, we get a duration: 


const auto toe (chrono::steady_clock::now()); 
return {s, toe - tic}; 

} 

9. Let's implement the actual program now. We loop until the user enters 
the input string correctly. In every loop step, we ask the user to please 
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enter the string "c++17"A and, then, call our get input function: 


int main() 

{ 

while (true) { 

cout << "Please type the word \"C++17\" as" 

" fast as you can.\n> 

const auto [user_ input, diff] = get_input(); 

10. Then we check the input. If the input is empty, we interpret this as a 
request to exit the whole program: 


if (user_input == "") { break; } 

/v 

11. If the user correctly typesA "c++i 7 ", we express our congratulations and 
then print the time the user needed to type the word correctly. 

/V /V 

TheA dif f. count () A method returns the number of seconds as a 
floating point number. If we had used the original STL seconds duration 
type, then we would have got a rounded A integer value, not a fraction. 
By feeding the milliseconds or microseconds constructor with our 
diff variable before calling count o, we get the same value transformed 
to a different unit: 


if (user_input == "C++17") { 

cout << "Bravo. You did it in:\n" 

<< fixed << setprecision(2) 

<< setw(12) << diff.count() 

<< " seconds.\n" 

<< setw(12) << milliseconds(diff) .count() 



<< " milliseconds.\n" 

<< setw(12) << microseconds(diff) .count () 
<< " microseconds.\n"; 
break; 


12. If the user has a typo in the input, we let him try again: 


} 


} else { 

cout << 

} 


"Sorry, your input does not match." 
" You may try again.\n"; 


13. Compiling and running the program leads to the following output. At 
first, with a typo, the program repeatedly asks for the correct input word. 
After typing the word correctly, it displays how long it took us to type it 
in three different time units: 

$ ./ratio_conversion 

Please type the word "C++17" as fast as you can. 

> c+17 

Sorry, your input does not match. You may try again. 

Please type the word "C++17" as fast as you can. 

> C++17 


Bravo. You did it in: 
1.48 seconds. 

1480.10 milliseconds. 
1480099.00 microseconds. 

How it works... 


While this section is all about converting between different time units, we 
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first had to choose one of the threeA available clock objects. There is 

/\ 

generally the choiceA between system ciock, steady ciock, and 
high resoiution ciock in the std:: chrono namespace. What are the 
differences between them? Let's have a closer look: 



Clock 


Characteristics 


This represents the system-wide real-time "wall" 
clock. It is the right choice if we want to obtain the 

system clock 

- local time. 


This clock is promised to be monotonic. This means 
that it will never be set back by any amount of time, 
steady ciock This can happen to other clocks when their time is 

corrected by minimal amounts, or even when the 
time is switched between winter and summer time. 

, . , , ^ . , This is the clock with the most fine-grained clock 

nigh resolution clock . . ° 

tick period the STL implementation can provide. 

Since we measured the time distance, or duration from one absolute point in 
time and the other absolute point in time (which we captured in the variables 
tic and toe), we are not interested if those points in time were globally 
skewed. Even if the clock was 112 years, 5 hours, 10 minutes, and 1 second 
(or whatever) late or ahead of time, then this does not make a difference on 
the difference between A them. The only important thing is that after we save 
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the time point ticA and before we save the time point toe, the clock must 
not be micro-adjusted (which happens on many systems from time to 

/V 

time)A because that would distort our measurement. For these requirements, 
steady ciock is the optimal choice. Its implementation can be based on the 

/V 

processor's timestampA counter, which always counts up monotonously since 
the system was started. 

Okay, now with the right time object choice, we are able to save points in 
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time viaA chrono:: steady ciock:: now () . The now function returns us a 
chrono : : time_point<chrono : : steady_clock> typed value. The difference 
between two such values (as in toe - tic) is a time span, or duration of type 
chrono: : duration. As this is the central type of this section, this gets a little 



complicated now. Let's have a closer look at the template type interface of 

duration: 

template< 

class Rep, 

class Period = std::ratio<l> 

> class duration; 

The parameters we can change are called Rep and Period. Rep is easy to 
explain: this is just the numeric variable type that is used to save the time 
value. For the existing STL time units, this is usually long long int. In this 
recipe, we chose double. Because of our choice, we can save time values in 
seconds by default and then convert them to milli- or microseconds. If we 
save the time duration of 1 . 2345 seconds in the chrono:: seconds type, then 
it would be rounded to one full second.A This way, we would have to save 
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the time difference between tic and tocA in chrono: :microsecondsA and 
could then convert to less-fine-grained units. With our double choice for Rep, 
we can convert up and down and lose only a minimal amount of precision, 
which does not hurt in this example. 

/V /V /V 

We usedA Rep = double for all our time units, soA theyA differed only in 

/V 

our choice of theA Period parameter: 


using seconds = chrono::duration<double>; 

using milliseconds = chrono::duration<double, 

ratio_multiply<seconds::period, milli»; 

using microseconds = chrono::duration<double, 

ratio_multiply<seconds::period, micro»; 


While seconds is the simplest unit to describe, as it works with Period = 

ratio<i>, the others have to be adjusted. As one millisecond is a thousandth 

/\ /\ 

of a second, we multiply the seconds: :periodA (which is justA a getter 
function to the Period parameter) with mini, which is a type alias for 
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std::ration, iooo>A (std: :ratio<a, b> represents the fractional value 

_ /v 

a/b). TheA ratiomuitipiy type is basically a compile time function, which 
represents the type that results from multiplying one ratio type with another. 

Maybe this sounds too complicated, so let's have a look at an example: 

/V 

ratio_multiply<ratio<2, 3>, ratio<4, 5>>A results in ratio<8, 

15>A because (2/3) * (4/5) = 8 / 15 . 



Our resulting type definitions are equivalent to the following definitions: 


using seconds = chrono::duration<double, ratio<l, 1»; 

using milliseconds = chrono::duration<double, ratio<l, 1000»; 

using microseconds = chrono::duration<double, ratiocl, 1000000>>; 

Having these types lined up, it is easy to convert between them. If we have a 
time duration d of type seconds, we can transform it to milliseconds just by 
feeding it through the constructor of the other type, that 

/V /V 

is,A milliseconds(d) .A 

There’s more... 


In other tutorials or books, you might run across duration cast whenever 
time durations are transformed. If we have a duration value of type 
chrono: :milliseconds and want to transform it to chrono: :hours, for 
example, we do indeed need to write duration_cast<chrono: :hours> 
(miiiiseconds vaiue) because these units depend on integer types. 
Transforming from fine-grained units to less-fine-grained units leads to 
precision loss in that case, which is why we need a duration cast. For 
double- or float-based duration units, this is not needed. 



Converting between absolute and relative 

/V 

times with A std::chrono 


Until C++11, it was quite a hassle to take the wall clock time and just print it, 
because C++ did not have its own time library. It was always necessary to 
call functions of the C library, which looks very archaic, considering that 
such calls could be encapsulated nicely into their own classes. 

Since C++11, the STL provides the chrono library, which makes time-related 
tasks much easier to implement. 

In this recipe, we are going to take the local time, print it, and play around by 
adding different time offsets, which is a really comfortable thing to do with 

std: : chrono. 


How to do it... 


We are going to save the current time and print it. Additionally, our program 
will add different offsets to the saved time point and print the resulting time 
points too: 

1. The typical include lines come first; then, we declare that we use 
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theA std namespace by default: 

#include <iostream> 

#include <iomanip> 

#include <chrono> 

using namespace std; 

2. We are going to print absolute time points. These will come along in the 

form of the chrono:: time point type template, so we will just overload 
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the output stream operator for it.A There are different ways to print the 
date and/or time part of a time point. We will just use the %c standard 





formatting. We could, of course, also print only the time, only the date, 
only the year, or whatever comes to our mind. All the conversions 
between the different types before we can finally apply put time look a 
bit clunky, but we are only doing this once: 


ostreams operator«(ostream &os, 

const chrono::time_point<chrono::system_clock 

{ 

const auto tt (chrono::system_clock::to_time_t(t)); 
const auto loct (std::localtime(&tt)); 
return os << put_time(loct, "%c"); 

} 

3. There are already STL type definitions for seconds, minutes, hours, 
and so on. We will add theA days type now. This is easy; we just have 
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to specialize the chrono: duration template by referring to hoursA and 
multiply with 2 4, because a full day has 24 hours: 


using days = chrono::duration< 
chrono::hours::rep, 

ratio_multiply<chrono::hours::period, ratio<24>>>; 

4. In order to be able to express a duration in multiples of days in the most 
elegant way, we can define our own days literal operator. Now, we can 
write 3_days to construct a value that represents threeA days: 


constexpr days operator ""_days(unsigned long long h) 

{ 

return days{h}; 

} 

5. In the actual program, we will take a time snapshot, which we simply 
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print afterward. This is very easy and comfortableA because we already 
implemented the right operator overload for this: 

int main() 

{ 

auto now (chrono::system_clock::now()); 

cout << "The current date and time is " << now << '\n'; 


/v /v 

6. Having saved the current time in theA nowA variable, we can add 



arbitrary durations to it and print those too. Let's add 12 hours to the 
current time and print what time we will have in 12 hours: 

chrono::hours chrono_12h {12}; 

cout << "In 12 hours, it will be " 

<< (now + chrono_12h)<< '\n 1 ; 

/v 

7. By declaring that we use theA chrono iiterais namespace by default, 
we unlock all the existing duration literals for hours, seconds, and so on. 
This way, we can elegantly print what time it was 12 hours and 15 
minutes ago, or 7 days ago: 


using namespace chrono_literals; 

cout << "12 hours and 15 minutes ago, it was " 

<< (now - 12h - 15min) << '\n 1 
<< "1 week ago, it was " 

<< (now - 7_days) << '\n 1 ; 

} 

8. Compiling and running the program yields the following output. 
Because we used %c as the format string for time formatting, we get a 
pretty complete description in a specific format. By playing around with 
different format strings, we can get it in any format we like. Note that 
the time format is not 12 hours AM/PM but 24 hours because the app is 
run on a European system: 

$ ./relative_absolute_times 

The current date and time is Fri May 5 13:20:38 2017 
In 12 hours, it will be Sat May6 01:20:38 2017 
12 hours and 15 minutes ago, it was Fri May5 01:05:38 2017 
1 week ago, it was Fri Apr 28 13:20:38 2017 


How it works... 


We obtained the current time point from std: : chrono :: system ciock. This 
STL clock class is the only one that can transform its time point values to a 

/V 

time structure that can beA displayed as a human-readable time description 
string. 



In order to print such time points, we implemented operator« for output 
streams: 

ostreams operator«(ostream &os, 

const chrono::time_point<chrono::system_clock 

{ 

const auto tt (chrono::system_clock::to_time_t(t)); 
const auto loct (std::localtime(&tt)); 
return os << put_time(loct, "%c"); 

} 

What happens here first, is that we transform from 

chrono::time_point<chrono::system_clock> to std::time_t. Values of 
this type can be transformed to a local wall clock relevant time value, which 
we do with std: : local time. This function returns us a pointer to a converted 
value (don't worry about the maintenance of the memory behind this 
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pointer;A it is a static object not allocated on the heap), which we can now 
finally print. 

The std: :put_time function accepts such an object together with a time 
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format string. "%c" displays a standard date-time string, such asA sun Mar 
12 11:33:40 20 17. A We could also have written "%m/%d/%y"; then the 
program would have printed the time in the format,A 03 / 12 / 17 . The whole 

list of existing format string modifiers for time is very long, but it is nicely 
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documented to its A full extent in the online C++ reference. 

Aside from printing, we also added time offsets to our time point. This is 

/v y\ 

very easyA because we can express time durations, such asA 12 hours and 15 

r-, A 

minutes as I2h + ismin. TheA chrono literals namespace already 
provides handy type literals for hours (h), minutes (min), seconds (s), 
milliseconds (ms), microseconds (us), and nanoseconds (ns). 

Adding such a duration value to a time point value creates a new time point 
value because these types have the right operator+ and operator- 
overloads, which is why it is so simple to add and display offsets in time. 



Safely signalizing failure with std:: optional 


When a program communicates with the outside world and relies on values it 
gets from there, then all kinds of failures can happen. 

This means that whenever we write a function that ought to return a value, 
but that can also possibly fail, then this must be reflected in some change of 
the function interface. We have several possibilities. Let's see how we 
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canA design the interface of a function that willA return a string, but that 
could also fail: 

• Use a success-indicating return value and output parameters: bool 

get_string(strings); 

• Return a pointer (or a smart pointer) that can be set to nuiiptr if there is 
a failure: string* get_string () ; 

• Throw an exception in the case of failure and leave the function 
signature very simple: string get_string () ; 

All these approaches have different advantages and disadvantages. Since 
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C++17, there is a new type thatA can be used to solve such a problem in a 
different way: std: optional. The notion of an optional value comes from 

purely functional programming languages (where they are sometimes called 
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Maybe types)A and can lead to very elegant code. 

/V 

We can wrap optional around our own types in order to signal emptyA or 
erroneous values. In this recipe, we will learn how to do that. 

How to do it... 

In this section, we will implement a program that reads integers from the 
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userA and sums them up. Because the user can always input random things 
instead of numbers, we will see how optional can improve our error 
handling: 



1. First, we include all the needed headers and declare that we use 
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theA stdA namespace: 

#include <iostream> 

#include <optional> 

using namespace std; 


/v 

2. Let's define an integer type, which,A maybe, contains a value. 

/V /V 

TheA std::optionaiA type does exactly that. By wrapping any type 
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into optional, we give it an additional possible state, whichA reflects 
that it currently has no value: 

using oint = optional<int>; 

3. By having defined an optional integer type, we can express that a 
function that usually returns an integer can also possibly fail. If we take 
an integer from user input, this can possibly fail because the user might 
not always enter an integer even though we asked him to do so. 
Returning an optional integer is perfect in this case. If reading an integer 
succeeds, we feed it into the optionai<int> constructor. Otherwise, we 
return a default constructed optional, which signals failure or emptiness: 

oint read_int() 

{ 

int i; 

if (cin >> i) { return {i}; } 

return {}; 

} 

4. We can do more than returning integers from functions that can 
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possiblyA fail. What if we calculate the sum of two optional integers? 
This can only lead to a real numeric sumA if both the operands contain 
an actual value. In any other case, we return an empty optional variable. 

This function needs a little more explanation: by implicitly transforming 
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the optionai<int> variables,A a and b, to boolean expressions (by 
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writing ! a and ! b), we get to know whetherA they contain actual values. 
If they do, we can access them like pointers or iterators by simply 
dereferencing them with *a and *b: 


oint operator+(oint a, oint b) 



{ 

if ( !a || !b) { return {}; } 

return {*a + *b}; 

} 

5. Adding a normal integer to an optional integer follows the same logic: 

oint operator+(oint a, int b) 

{ 

if ( !a) { return {}; } 

return { *a + b}; 

} 

/v 

6. Let's now write a program thatA does something with optional integers. 
We let the user enter two numbers: 

int main() 

{ 

cout << "Please enter 2 integers.\n> 

auto a {read_int()}; 
auto b {read int()}; 


7. Then we add those input numbers and additionally add the value 10 to 
their sum. Since a and b are optional integers, sum will also be an 
optional integer type variable: 


auto sum (a + b + 10); 

/v /v 

8. If a A and/or b do A not contain a value, then sum cannot possibly contain 

a value either. The nice thing about our optional integers now is that we 
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do not need to explicitly check a and b.A What happens when we sum 
up empty optionals is perfectly sane and defined behavior because we 
defined operator+ in a safe way for those types. This way, we can 
arbitrarily add many possibly empty optional integers, and we'll only 
need to check the resulting optional value. If it contains a value, then we 
can safely access and print it: 


if (sum) { 

cout << *a << " + " << *b << " 
<< *sum << '\n'; 


If 


+ 10 
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9. If the user enters A something non-numeric, we error out: 


} 


} else { 
cout << 

} 


sorry, the input was " 

something else than 2 numbers.\n"; 


10. That's it. When we compile and run the program, we get the following 
output: 

$ ./optional 

Please enter 2 integers. 

>12 

1 + 2 + 10 = 13 


11. Running the program again and entering something non-numeric yields 
the error message we prepared for this case: 

$ ./optional 

Please enter 2 integers. 

> 2 z 

sorry, the input was something else than 2 numbers. 


How it works... 


Working with optional is generally very simple and convenient. If we want 
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to attach the notion ofA possible failure or optionality to any type t, we can 
just wrap it into std: :optionai<T> and that's it. 

Whenever we get such a value from somewhere, we have to check 

/V /V /V 

whether A it is in the emptyA state or whether A it contains a real value. 

TheA booiA optional :: hasvaiue () function does that for us. If it returns 
true, we may access the value. Accessing the value of an optional can be 
done with T& optional:: value () . 

Instead of always writing if (x. has_vaiue () ) {...} and x . value ( ), we 

~ /V /V 

can also write if (x) {.. .} and *x. TheA std: :optionaiA type defines 
explicit conversion to bool and operator* in such a way that dealing with an 

/V 

optional type is similar toA dealing with a pointer. 



/V 

Another handy operator helper thatA is good to know is the operators 
Overload of optional. If we have a struct Foo { int a; string b; } type 
and want to access one of its members through an optionai<Foo> 

/V 

variable,A x, then we can write x->a or x->b. Of course, we should first 
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check whether A x actually has a value. 

If we try to access an optional value even though it does not have a value, 
then it will throw std:: logic error. This way, it is possible to mess around 
with a lot of optional values without always checking them. Using a try- 
catch clause, we could write code in the following form: 

cout << "Please enter 3 numbers:\n"; 
try { 

cout << "Sum: " 

<< (*read_int() + *read_int() + *read_int()) 

<< '\n'; 

} catch (const std::bad_optional_access &) { 

cout << "Unfortunately you did not enter 3 numbers\n"; 

} 

Another gimmick of std: optional is optional:: vaiue_or. If we want to 
take an optional's value and fall back to a default value if it is in the empty 
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state, A then this is of help, x = optionaivar. vaiueor (123) doesthisjob 
in one concise line, where 123 is the fallback default value. 



Applying functions on tuples 


Since C++11, the STL provides std:: tuple. This type allows us to 
sporadically bundle multiple values into a single variable and reach 

A _ 

themA around. The notion of tuples has been there for a long time in a lot of 

programming languages, and some recipes in this book are already devoted to 

/\ 

this type A because it is extremely versatile to use. 

/V 

However, we sometimes end up with values bundled up in a tupleA and then 
need to call functions with their individual members. Unpacking the members 
individually for every function argument is very tedious (and error-prone if 
we introduce a typo somewhere). The tedious form looks like this: 

func(get<0>(tup), get<l>(tup), get<2>(tup), ...);. 

/v 

In this recipe, youA will learn how to pack and unpack values to and from 
tuples in an elegant way, in order to call some functions that don't know 
about tuples. 

How to do it... 


/V 

We are going to implement a program thatA packs and unpacks values to and 
from tuples. Then, we will see how to call functions that know nothing about 
tuples with values from tuples: 

1. First, we include a lot of headers and declare that we use 
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theA stdA namespace: 

#include <iostream> 

#include <iomanip> 

#include <tuple> 

#include <functional> 

#include <string> 

#include <list> 


using namespace std; 




2. Let's first define a function that takes multiple parameters describing a 
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student and prints them. AA lot of legacy- or C-function interfaces look 
similar.: 


static void print_student(size_t id, const string Sname, do 
{ 

cout << "Student " << quoted(name) 

« ", ID: " « id 

<< ", GPA: " << gpa << 1 \n 1 ; 

} 

/v 

3. In the actual program, we define a tuple type on the flyA and fill it with 
meaningful student data: 


int main() 

{ 

using student = tuple<size_t, string, double>; 
student john {123, "John Doe"s, 3.7}; 

4. In order to print such an object, we can decompose it to its individual 
members and call print student with those individual variables: 

{ 

const auto &[id, name, gpa] = john; 
print_student(id, name, gpa); 

} 

cout << "-\n"; 

/v 

5. Let's create a whole setA of students in the form of an initializer list of 
student tuples: 

auto arguments_for_later = { 

make_tuple(234, "John Doe"s, 3.7), 
make_tuple(345, "Billy Foo"s, 4.0), 
make_tuple(456, "Cathy Bar"s, 3.5), 

}; 


6. We can still relatively comfortably print them all, but in order to 
decompose the tuple, we need to care how many elements such tuples 
have. If we have to write such code, then we will also have to restructure 
it in case the function call interface changes: 


for (const auto &[id, name, gpa] : arguments for later) 




print_student(id, name, gpa); 

} 

cout << "-\n"; 

7. We can do better. Without even knowing the argument types of 
printstudent or the number of members in a student tuple, we can 
directly forward the tuple's content to the function using std:: apply. 
This function accepts a function pointer or a function object and a 

/V 

tupleA and then unpacks the tuple in order to call the function with the 
tuple members as parameters: 

apply(print_student, john); 
cout << "-\n"; 

8. This also works nicely in a loop, of course: 


for (const auto &args : arguments_for_later) { 

apply(print_student, args); 

} 

cout << "-\n"; 

} 

9. Compiling and running the program shows that both ways work, as we 
assumed: 

$ ./apply_functions_on_tuples 


Student 

"John Doe", ID: 

123, GPA: 

3.7 

Student 

"John Doe", ID: 

234, GPA: 

3.7 

Student 

"Billy Foo", ID: 

345, GPA: 

4 

Student 

"Cathy Bar", ID: 

456, GPA: 

3.5 

Student 

"John Doe", ID: 

123, GPA: 

3.7 

Student 

"John Doe", ID: 

234, GPA: 

3.7 

Student 

"Billy Foo", ID: 

345, GPA: 

4 

Student 

"Cathy Bar", ID: 

456, GPA: 

3.5 


How it works... 


The std:: apply is a compile-time helper that helps us work more agnostic 










about the types we handle in our code. 


/V 

Imagine we have a tuple t A with the values (123, "abc"s, 456.0). This 

/V 

tuple has the type,A tupie<int, string, doubiex Additionally, assume 

/\ /\ /v 

that we have a functionA f A with the signatureA int f (int, string, 
double) (the types can also be references). 

Then, we can write x =A apply (f, t ), which will result in a function 

/V _ /V /\ 

call, A x =a f ( 123 , "abc"s, 456.0). The A appiyA method does even 
return to us what f returns. 



Quickly composing data structures with 
std:: tuple 


Let's have a look at a basic use case for tuples that we most probably already 

/V 

know. We can defineA a structure as follows, in order to just bundle some 
variables: 


struct Foo { 
int a; 
string b; 
float c; 

}; 

Instead of defining a structure as in the preceding example, we can also 
define a tuple: 


using Foo = tuple<int, string, float>; 


We can access its items using the index number of the type from the type list. 

/\ /\ 

In order to access the first member of a tuple, A t,A we can use std:: get<o> 

/V /V 

(t) A to access the second member we writeA std:: get<i>, and so on. If the 
index number is too large, then the compiler will even safely error out. 

Throughout the book, we have already used the decomposition capabilities of 
C++17 for tuples. They allow us to decompose a tuple quickly by just writing 

/V 

auto [a, b, c] = sometuple A in order to access its individual items. 

/v 

Composing and decomposing single data structures areA not the only things 
we can do with tuples. We can also concatenate or split tuples, or do all kinds 
of magic. In this recipe, we will play around with such capabilities in order to 
learn how to do it. 


How to do it... 

In this section, we will write a program that can print any tuple on the fly. In 





addition to that, we will write a function that can zip tuples together: 


1. We need to include a number of headers firstA and then we declare that 

/\ /\ 

we use the A stdA namespace by default: 

#include <iostream> 

#include <tuple> 

#include <list> 

#include <utility> 

#include <string> 

#include <iterator> 

#include <numeric> 

#include <algorithm> 

using namespace std; 


2. As we will be dealing with tuples, it will be interesting to display their 
content. Therefore, we will now implement a very generic function that 
can print any tuple that consists of printable types. The function accepts 

/V 

an output stream reference os, whichA will be used to do the actual 

/\ 

printing, and a variadic argument list, whichA carries all the tuple 

members. We decompose all the arguments into the first element and 

/\ 

put it into the argument, A v, and the rest, which is stored in the 
argument pack vs... : 

template <typename T, typename ... Ts> 

void print_args(ostream &os, const T &v, const Ts &...vs) 

{ 

os << v; 

/v 

3. If there are arguments left in the parameter pack,A vs, these are printed 

interleaved with ", " using the initialize r_i i s t expansion trick. 

/\ 

YouA learned about this trick in the Chapter 4 . Lambda Expressions : 

(void)initializer_list<int>{ ((os << ", " << vs), 0) ... 

} 

4. We can now print arbitrary sets of arguments by writing 

print_args (cout, 1, 2 , "too", 3, "bar "), for example. But 
/\ 

thisA has nothing to do with tuples yet. In order to print tuples, we 



/\ /\ 

overload the stream output operator A «A for any case of tuples by 
implementing a template function that matches on any tuple 
specialization: 

template <typename ... Ts> 

ostreams operator«(ostream &os, const tuple<Ts...> &t) 

{ 

5. Now it gets a little complicated. We first use a lambda expression that 

arbitrarily accepts many parameters. Whenever it is called, it prepends 

/\ 

the osA argument to those argumentsA and then calls print args with 
the resulting new list of arguments. This means that a call to 

capt_tup(...some parameters...) leads to aA print_args(os, 

/v 

. . . some parameters . . .) A call: 

auto print_to_os ([&os] (const auto &...xs) { 

print_args(os, xs...); 

}) ; 

6. Now we can do the actual tuple unpacking magic. We use std:: apply 
to unpack the tuple. All values will be taken out of the tuple then and 
lined up as function arguments for the function that we provide as the 

_ A 

first argument. This just means that if we have a tuple, A t = ( 1 , 2 , 

3), and call apply (capt_tup, t), then this will lead to a function 

/\ /V 

call,A capt tup ( 1 , 2 , 3 ), which in turn leads to theA function 

/V ~ 

call,A print_args (os, l, 2 , 3). This is just what we need. As a nice 
extra, we surround the printing with parentheses: 

os « " ("; 

apply(print_to_os, t); 
return os << 

} 

7. Okay, now we wrote some complicated code that will make our life 
much easier when we want to print a tuple. But we can do a lot more 
with tuples. Let's, for example, write a function that accepts an iterable 

/V 

range, such asA a vector or a list of numbers, as an argument. This 

/V 

function willA then iterate over that range and then return us the sum of 
all the numbers in the range and bundle that with the minimum of all 
values, the maximum of all values, and the numeric average of them. By 



/V 

packing these fourA values into a tuple, we can return them as a single 
object without defining an additional structure type: 


template <typename T> 

tuple<double, double, double, double> 
sum_min_max_avg(const T &range) 

{ 

_ A 

8 . The A std:: minmax element function returns us a pair of iterators that 
respectively point to the minimum and maximum values of the input 
range. TheA std:: accumuiateA method sums up all the values in its 
input range. This is all we need to return the four values thatA fit in our 
tuple! 


} 


auto min_max (minmax_element(begin(range), end(range))) 
auto sum (accumulate(begin(range), end(range), 0.0) 

return {sum, *min_max.first, *min_max.second, 
sum / range.size ()}; 


9. Before implementing the main program, we will implement one last 

/\ 

magic helper function. I call it magicA because it really looks 

complicated at first, but after understanding how it works, it will turn 

/\ 

out as a really slick and nice helper. It willA zip two tuples. This means 

/V /\ 

that if we feed it a tuple, A ( l , 2 , 3 ), and another tuple, A (' a ', ' b ', 

' c ' ) , it will return a tuple (l, 'a', 2, ' b', 3, ' c'): 

template <typename Tl, typename T2> 
static auto zip (const Tl &a, const T2 &b) 

{ 


10. Now we arrived at the most complex lines of code of this recipe. We 

/V /\ 

create a function object,A z, which acceptsA an arbitrary number of 
arguments. It then returns another function object that captures all these 

/\ /V 

arguments in aA parameter pack,A xs, but also accepts another arbitrary 
number of arguments. Let's sink this in for a moment. Wit hin this inner 

/V 

function object, we can accessA both lists of arguments in the form of 

/\ 

the parameter packs,A xs and ys. And now let's have a look what we 
actually do with these parameter packs. The 

/V 

expression,A make tuple (xs, ys) ..., groups the parameter packs item 



wise. This means that if we have xs = 1 , 2 , 3 and ys = ' a', ' b', 

/V 

' c', this will result in a new parameter pack, A ( 1 , 'a'), (2, ' b'), 

( 3 , ' c'). This is a comma-separated list of three tuples. In order to get 
them all grouped in one tuple, we use std:: tupie cat, which accepts 
an arbitrary number of tuples and repacks them into one tuple. This way 
we get a nice ( 1 , 'a', 2 , 'b\ 3 , 1 c') tuple: 

auto z ([](auto ...xs) { 

return [xs...] (auto ...ys) { 

return tuple_cat(make_tuple(xs, ys) ...); 

} ; 

}) ; 

/V 

11. The last step is unwrapping all the values from the input tuples,A a and 

/v 

b, and pushing them into z. The expression,A apply (z, a), puts all the 

/V 

values from a into the parameter pack xs, andA apply (..., b) puts all 
the values of b into the parameter pack ys. The resulting tuple is the 
large zipped one, which we return to the caller: 

return apply(apply(z, a), b); 

} 

12. We invested a considerable amount of lines into helper/library code. 
Let's now finally put it to use. First, we construct some arbitrary tuples. 

/V /V 

TheA student containsA ID, name, and GPA score of a student. 

/V 

TheA student desc contains strings that describe what those fields 

_ ^ 

mean in human-readable form. TheA std: :make_tupie is a really nice 

/V 

helperA because it automatically deduces the type of all the arguments 
and creates a suitable tuple type: 

int main() 

{ 

auto student_desc (make_tuple("ID", "Name", "GPA")); 
auto student (make_tuple(123456, "John Doe", 3.7)) 

/v 

13. Let's just print what we have. This is really simpleA because we just 
implemented the right operator« overload for that: 

cout << student_desc << '\n' 

<< student << '\n'; 



14. We can also group both the tuples on the fly with std:: tupie cat and 
print them like this: 

cout << tuple_cat(student_desc, student) << '\n'; 

15. We can also create a new zipped tuple with our zip function and also 
print it: 


auto zipped (zip(student_desc, student)); 
cout << zipped << '\n 1 ; 

16. Let's not forget our sum min max avg function. We create an initializer 

/V 

list that contains some numbersA and feed it into this function. To make 
it a little bit more complicated, we create another tuple of the same size, 

/V 

whichA contains some describing strings. By zipping these tuples, we 
get a nice, interleaved output, as we will see when we run the program: 

auto numbers = {0.0, 1.0, 2.0, 3.0, 4.0}; 
cout << zip( 

make_tuple("Sum", "Minimum", "Maximum", "Averag 
sum_min_max_avg(numbers)) 

« '\n T ; 

} 

17. Compiling and running the program yields the following output. The 
first two lines are just the individual student and student desc tuples. 

/V /V 

Line 3A is the tuple composition we got by usingA tupie cat. Line 4 
contains the zipped student tuple. In the last line, we see the sum, 
minimum, maximum, and average value of the numeric list we last 
created. Because of the zipping, it is really easy to see what each value 
means: 

$ ./tuple 
(ID, Name, GPA) 

(123456, John Doe, 3.7) 

(ID, Name, GPA, 123456, John Doe, 3.7) 

(ID, 123456, Name, John Doe, GPA, 3.7) 

(Sum, 10, Minimum, 0, Maximum, 4, Average, 2) 


How it works... 



Some of the code in this section is admittedly complicated. We wrote an 
operator« implementation for tuples, which looks very complex but 
supports all kinds of tuples that themselves consist of printable types. Then 

/V /\ /\ 

we implemented theA sum_min_max_avgA function, whichA just returns a 

/y 

tuple. Another very complicated thing to get ourA head around was the 
function zip. 

The easiest part wasA sum min max avg. The point about it is that when we 

/\ /V /V 

define a function that returns an instance A tupie<Foo,ABar,ABaz> f(), 
we Can JUSt write return { foo_instance, bar_instance, baz_instance}; 

in that function to construct such a tuple. If you have trouble understanding 
the STL algorithms we used in theA sum_min_max_avg function, then you 

A _ 

might want to have a look atA the Chapter 5 . STL Algorithm Basics of this 
book, where we already had a closer look at them. 

The other code was so complicated that we dedicate the specific helpers their 
own subsections: 

operator« for tuples 

Before we even touched operator« for output streams, we implemented 
theA print argsA function. Due to its variadic argument nature, it accepts 
any number and type of arguments, as long as the first one is an ostream 
instance: 

template <typename T, typename ... Ts> 

void print_args(ostream &os, const T &v, const Ts S...vs) 

{ 

os << v; 

(void)initializer_list<int>{((os << ", " << vs), 0)...}; 

} 

This function prints the first item, A v, and then prints all the other items from 

/\ 

the parameter pack,A vs. We print the first item individually because we 
want to have all items interleaved with ", " but we do not want this string 
leading or trailing the whole list (as in " 1 , 2 , 3 , " or ", 1 , 2 , 3"). We 

/V 

learned about the initiaiizer iist expansion trick in Chapter 4 . A Lambda 
Expressions , in the recipe Calling multiple functions with the same input. 




Having that function lined up, we have everything we need in order to print 
tuples. Our operator« implementation looks as follows: 

template <typename ... Ts> 

ostreams operator«(ostream &os, const tuple<Ts...> &t) 

{ 

auto capt_tup ([&os](const auto &...xs) { 

print_args(os, xs . . . ) ; 

}) ; 

os « " ("; 

apply(capt_tup, t); 

return os << 

} 

The first thing we do is defining the function object,A capt tup. When we 
Call capt_tup (too, bar, whatever), this results in 

/v 

theA call,A print_args (os , too, bar, whatever). The Only thing this 

/\ 

function object does A is prepend the output stream object os to its variadic 
list of arguments. 

Afterward, we use std: : apply in order to unpack all the items from tuple t. 
If this step looks too complicated, please have a look at the recipe before this 

/V /V 

one, which is dedicated to demonstrating howA std:: appiyA works. 


The zip function for tuples 

, /v 

The zip function accepts twoA tuples, but looks horribly complicated, 
although it has a very crisp implementation: 

template ctypename Tl, typename T2> 
auto zip(const Tl &a, const T2 &b) 

{ 

auto z ([](auto ...xs) { 

return [xs...] (auto ...ys) { 

return tuple_cat(make_tuple(xs, ys) ...); 

}; 

}) ; 

return apply(apply(z, a), b); 


In order to understand this code better, imagine for a moment that the tuple a 



/v y\ 

carries the values, A 1 , 2 , 3 , and tuple b carries the values, A ' a', ' b', 

' C ' . 

/V 

In such aA case, calling apply (z, a) leads to a function call z ( 1 , 2 , 3 ), 

/V 

which returns another function object that captures those values,A 1 , 2 , 3 , 

in the parameter pack xs. When this function object is then called with 

/\ 

apply (z(i, 2 , 3), b), it gets the values,A 'a', 'b\ 'c', stuffed into the 

/v _ 

parameter pack,A ys. This is basically the same as if we called z ( 1 , 2 , 3 ) 
Ca', ' b', ' c') directly. 

Okay, now that we have xs = ( 1 , 2 , 3 ) and ys = (' a', ' b', ' c'), what 

_ A 

happens then? The expression A tupie cat (make tuple (xs, ys) ...) does 
the following magic; have a look at the diagram: 



(1, 'a', 2, 'b\ 3, 'c') 


At first, the items from xs and ys are zipped together by interleaving them 
pairwise. This "pairwise interleaving" happens in the make_tupie (xs, ys) 

... expression. This initially only leads to a variadic list of tuples with two 

items each. In order to get one large tuple, we apply tupie cat on them and 

/\ 

then we finally getA a large concatenated tuple that contains all the members 
of the initial tuples in an interleaved manner. 



Replacing void* with std::any for more type 
safety 


It can happen that we want to store items of any type in a variable. For such a 

/V /V 

variable, we then need to be able to check whetherA it containsA anything, 

/\ 

and if it does, we need to be able to distinguish what A it contains. All this 
needs to happen in a type-safe manner. 

In the past, we were basically able to store pointers to various objects in a 

/V 

void* pointer. AA void typed pointer alone cannot tell us what kind of object 

/\ 

it points to, so we would need to handcraft some A kind of additional 

/V 

mechanism thatA tells us what to expect. Such code quickly leads to quirky 
looking and unsafe code. 

_ /V /V 

Another addition of C++17 to the STL is theA std:: anyA type. It is designed 

/V 

to hold variables of any kind and provides facilities that enable forA type- 
safe inspection and access to it. 

In this recipe, we will play around with this utility type in order to get a 

/V 

feeling ofA it. 

How to do it... 


We will implement a function that tries to be able to print everything. It uses 
std:: any as its argument type: 

1. First, we include some necessary headers and declare that we use 

/\ /V 

theA stdA namespace: 

#include <iostream> 

#include <iomanip> 

#include <list> 

#include <any> 

#include <iterator> 




using namespace std; 


2. In order to reduce the number of angle bracket syntax in the following 
program, we define an alias for iist<int>, which we will use later: 

using int_list = list<int>; 

3. Let's implement a function that claims to be able to print anything. The 

promise is that it prints anything provided as an argument in the form of 

/\ 

an std:: anyA variable: 


void print_anything(const std::any &a) 

{ 

4. The first thing we need to check is if the argument contains anything or 
if it is just an empty any instance. If it is empty, then there is no sense in 
trying to figure out how to print it: 

if (!a.has_value()) { 

cout << "Nothing.\n"; 

5. If it is not empty, we can try to compare it with different types until we 

/v 

see a match. The first type to try isA string. If it is a string, we can 
cast a to a string typed reference using std: :any_cast and just print it. 
We put the string in quotes for cosmetic reasons: 

} else if (a.type() == typeid(string)) { 

cout << "It's a string: " 

<< guoted(any_cast<const string&>(a)) << '\n'; 

/v 

6 . If it is not a string, it might be an int. A In case this type matches, we 
can use any_cast<int> to obtain the actual int value: 

} else if (a.type() == typeid(int)) { 
cout << "It's an integer: " 

<< any_cast<int>(a) << '\n'; 

/v 

7. std :: anyA does not only work with such simple types as string and 
int. We can also put a whole map or list or whatever composed 
complex data structure into an any variable. Let's see if the input is a list 
of integers, and if it is, we can just print it like we would print a list: 



} else if (a.type() == typeid(int_list)) { 

const auto &1 (any_cast<const int_list&>(a)); 

cout << "It's a list: 
copy(begin(1), end(l), 

ostream_iterator<int>{cout, ", "}); 
cout << ' \n' ; 

8 . If none of these types match, we run out of type guesses. Let's give up in 
that case and tell the user that we have no idea how to print this: 

} else { 

cout << "Can't handle this item.\n"; 

} 

} 

9. In the main function, we can now call this function with arbitrary types. 

/V /V 

We can call it with an empty any variable using { } A or feed it withA a 
string "abc" or an integer. Because std: :any can be constructed from 
such types implicitly, there is no syntax overhead. We can even 
construct a whole list and throw it into this function: 


int main() 

{ 

print_anything({}); 
print_anything("abc"s); 
print_anything(123); 
print_anything(int_list{1, 2, 3}); 

10. If we are going to put objects that are really expensive to copy into an 
any variable, we can also perform an in-place construction. Let's try this 

/V /v 

with our list type. TheA in_piace_type_t<int_iist>{ }A expression is 
an empty object that gives the constructor of any enough information to 
know what we are going to construct. The second parameter,A {l, 2 , 

3}, is just an initializer list that A will be fed to the intiist embedded 
in the any variable for construction. This way, we avoid unnecessary 
copies or moves: 

print_anything(any(in_place_type_t<int_list>{}, {1, 2, 

} 


11. Compiling and running the program yields the following output, which 



is just what we expected: 


$ ./any 
Nothing. 

It's a string: "abc" 
It's an integer: 123 
It's a list: 1, 2, 3, 
It's a list: 1, 2, 3, 


How it works... 

/v 

The std:: anyA type is similar in one regard to std: optional— it has a 
has vaiue () method that tells if an instance carries a value or not. But apart 
from that, it can contain literally anything, so it is more complex to handle 

/V 

compared withA optional. 

/v 

Before accessing the content of an any variable, weA need to find out what 

/\ 

type it carries and, then,A cast it to that type. 

Finding out if an any instance holds a type t value can be done with a 
comparison: x . type () == typeid (T) . If this comparison results in true, 
then we can use any cast to get at the content. 

Note that any_cast<T> (x) returns a copy of the internal t value in x. If we 
want a reference in order to avoid copying of complex objects, we need to 
use any_cast<T&> (x) . This is what we did when we accessed the internal 

/V 

string or iist<int>A objects in this section's code. 

Note 

If we cast an instance of any to the wrong type, it will throw an 

std: :bad_any_cast exception. 



Storing different types with std:: variant 


There are not only struct and ciassA primitives in C++ that enable us to 
compose types. If we want to express that some variable can hold either some 
type a or a type b (or c, or whatever), we can use union. The problem with 
unions is that they cannot tell us they were actually initialized to which of the 
types that they can hold. 

Consider the following code: 

union U { 

int a; 
char *b; 
float c; 

}; 


void func (U u) { std::cout << u.b << '\n'; } 

/V /V 

If we call theA funcA function with a union that was initialized to hold an 

/V 

integer via memberA a, there is nothing that prevents us from accessing it, as 
if it was initialized to store a pointer to a string via member b. All kin ds of 
bugs can be spread from such code. Before we start to pack our union with an 
auxiliary variable that tells us to what it was initialized in order to gain some 
safety, we can directly use std:: variant, which came with C++17. 

/v 

TheA variant is kind of the new-school, type-safe, and efficient union type. 
It does not use the heap, so it is as space-efficient and time-efficient as a 
union-based handcrafted solution could be, so we do not have to implement it 
ourselves. It can store anything apart from references, arrays, or 

/V /V 

theA voi dA type. 

In this recipe, we will construct an example that profits from variant in 

/\ 

order to get a feeling ofA how to use this cool new addition to the STL. 


How to do it... 




/V 

Let's implement a program that knows the types,A cat and dog, and that 

/\ 

stores a mixed list of cats and dogs without using anyA runtime polymorphy: 


1. First, we include all the needed headers and define that we use 

/V /V 

theA stdA namespace: 

#include <iostream> 

#include <variant> 

#include <list> 

#include <string> 

#include <algorithm> 

using namespace std; 

/v /v 

2. Next, we implement two classes that A have similar functionalityA but 
are not related to each other in any other way, in contrast to classes 

/V 

that,A for example, inherit from the same interface or a similar interface. 

/V 

The first class is cat. A catA object has a name and can say meow. 

class cat { 

string name; 

public: 

cat (string n) : namefn} {} 

void meow() const { 

cout << name << " says Meow!\n"; 

} 

}; 

/v /v 

3. The other class is dog. A dogA object does not say meow A but woof, of 
course: 



} 



/V 

4. Now we can define anA animal type, which is just a type alias to 
std:: variant<dog, cat>. This is basically the same as an old-school 
union but has all the extra features that variant provides: 

using animal = variant<dog, cat>; 


5. Before we write the main program, we implement two helpers first. One 
helper is an animal predicate. By calling is_type<cat> (...) or 
is_type<dog> (...), we can find out if an animal variant instance 

/V 

holdsA a cat or a dog. The implementation just calls 

hoids aiternative, which is a generic predicate function for variant 

types: 


template <typename T> 

bool is_type(const animal &a) { 

return holds_alternative<T>(a); 

} 

_ /\ 

6 . The second helper is a structure that A acts as a function object. It is a 

/\ 

twofold function objectA because it implements operator () twice. One 
implementation is an overload that accepts dogs and the other accepts 
cats. For these types, it just calls the woof or the meow function: 


struct animal_voice 
{ 

void operator()(const dog &d) const { d.woof(); } 
void operator()(const cat &c) const { c.meow(); } 

}; 


7. Let's put these types and helpers to use. First, we define a list of animal 
variant instances and fill it with cats and dogs: 

int main() 

{ 

list<animal> 1 {cat{"Tuba"}, dog{"Balou"}, cat{"Bobby"} 

/v 

8 . Now, we print the contents of theA list three times, and each time in a 

different way. One way is using variant::index() . 

/\ /\ /\ 

BecauseA animaiA is an alias ofA variant<dog, cat>, a return value 
of o means that the variant holds a dog instance. Index l means it is a 
cat. The order of the types in the variant specialization is the key here. 



In the switch case block, we access the variant with get<T> in order to 
get the actual cat or dog instance inside: 

for (const animal &a : 1) { 

switch (a.index()) { 

case 0: 

get<dog> (a) .woof (); 
break; 
case 1: 

get<cat> (a) .meow(); 
break; 

} 

} 

cout << "-\n"; 

9. Instead of using the numeric index of the type, we can also explicitly ask 

_ /V 

for every type. TheA get_if<dog> returns a dog-typed pointer to the 
internal dog instance. If there is no dog instance inside, then the pointer 
is null. This way, we can try to get at different types until we finally 
succeed: 


for (const animal &a : 1) { 

if (const auto d (get_if<dog>(&a)); d) { 
d->woof(); 

} else if (const auto c (get_if<cat> (&a)); c) { 
c->meow(); 

} 

} 

cout << "-\n"; 

10. The last and most elegant way is variant::visit. This function accepts 
a function object and a variant instance. The function object must 
implement different overloads for all the possible types the variant can 

/V 

hold. We implemented a structure with the rightA operator ( ) overloads 
before, so we can use it here: 


for (const animal &a : 1) { 

visit(animal_voice{}, a); 

} 

cout << "-\n"; 

11. At last, we will count the number of cats and dogs in the variant list. The 
is_type<T> predicate can be specialized on cat and dog and can then be 






used in combination with std:: count if to return us the number of 
instances of this type: 

cout << "There are " 

<< count_if(begin(1), end(l), is_type<cat>) 

<< " cats and " 

<< count_if(begin(1), end(l), is_type<dog>) 

<< " dogs in the list.\n"; 

} 

12. Compiling and running the program first yields the same list printed 

/V /V 

threeA times. After that,A we see that the is type predicates combined 
with count if work just fine: 

$ ./variant 
Tuba says Meow! 

Balou says Woof! 

Bobby says Meow! 


Tuba says Meow! 
Balou says Woof! 
Bobby says Meow! 


Tuba says Meow! 
Balou says Woof! 
Bobby says Meow! 


There are 2 cats and 1 dogs in the list. 


How it works... 

/V 

The std:: variant A type is kind of similar to std:: any because both can 
hold objects of different types, and we need to distinguish at runtime what 
exactly they hold before we try to access their content. 

On the other hand, std:: variant is different from std:: any in the regard 
that we must declare what it shall be able to store in the form of a template 
type list. An instance of std:: variant<A, b, c> must hold one instance of 

/v _ 

typeA a, b, or c. There is no possibility to hold none of them, which means 






that std:: variant has no notion of optionality. 

/\ 

A variant of type,A variant<A, b, c>, mimics a union type that could look 
like the following: 

union U { 

A a; 

B b; 

C c; 

}; 


The problem with unions is that we need to build our own mechanisms to 
distinguish if it was initialized with an a, b, or c variable. 

TheA std:: variant A type can do this for us without much hassle. 

/v 

In the code inA this section, we used three different ways to handle the 
content of a variant variable. 

The first way was the index () function of variant. For a variant type 
variant<A, b, c> it can return index o if it was initialized to hold an a type, 
or 1 for b, or 2 for c, and so on for more complex variants. 

The next way isA the get_if<T> function. It accepts the address of a variant 

/V /V 

object and returns a T-typed pointer to its content. If theA tA type is wrong, 
then this pointer will be a null pointer. It is also possible to call get<T> (x) 
on a variant variable in order to get a reference to its content, but if that does 
not succeed, this function throws an exception (before doing such get-casts, 
checking for the right type can be done with the Boolean predicate 

holds_alternative<T>(x)). 

The last way to access the variant is the std:: visit function. It accepts a 

_ /V /V 

function object and a variant instance. TheA vis itA function then checks 
of which type the content of the variant is and then calls the right operator () 
overload of the function object. 

For exactly this purpose, we implemented the animai voice type because it 
can be used in combination with visit and variant<dog, cat>: 

struct animal_voice 
{ 



void operator () (const dog &d) const { d.woof(); } 
void operator()(const cat &c) const { c.meow(); } 


The visit- way of accessing variants can be considered the most elegant 

/\ 

oneA because the code sections that actually access the variant do not need to 
be hardcoded to the types the variant can hold. This makes our code easier to 
extend. 


Note 


The claim that a variant type cannot hold no value was not 

/V 

completely true. By adding theA std::monostate type to its type 
list, it can indeed be initialized to hold no value. 



Automatically handling resources with 
std::unique_ptr 


Since C++11, the STL provides smart pointers that really help keep track of 

/V 

dynamic memory and its disposal. Even before C++11, there was aA class 

/V 

calledA auto ptr that was already able to do automatic memory disposal, 

/V 

but itA was easy to use the wrong way. 

However, with the C++11-generation smart pointers, we seldom need to 
write new and delete ourselves, which is a really good thing. Smart pointers 

/V 

areA a shiny example of automatic memory management. If we maintain 

/V 

dynamically allocated objects with unique ptr, we are basicallyA safe from 
memory leaks, because upon its destruction this class automatically calls 
delete on the object it maintains. 

/V 

A unique pointer expresses ownership of the object it points toA and follows 
its responsibility of freeing its memory again if it is no longer used. This class 

/V 

has the potential of relieving us foreverA from memory leaks (at least 
together with its companions shared ptr and weak ptr, but in this recipe, 

/v 

we solely concentrate on unique ptr). And the best thing isA that it imposes 

/V 

no overhead on space and runtime performance, compared withA code with 

/\ 

raw pointersA and manual memory management. (Okay, it still sets its 
internal raw pointer to nuiiptr internally after destruction of the object it 
points to, which cannot always be optimized away. Most manually written 
code that manages dynamic memory does the same, though.) 

/V 

In this recipe, we will a look at unique ptrA and how to use it. 

How to do it... 

We will write a program that shows us how unique ptr handles memory by 
creating a custom type that adds some debug messages upon its construction 
and destruction. Then, we will play around with unique pointers, maintaining 




dynamically allocated instances of it: 


1. First, we include the necessary headers and declare that we use 

/\ /V 

theA stdA namespace: 

#include <iostream> 

#include <memory> 

using namespace std; 


2. We are going to implement a little class for the object we are going to 
manage using unique ptr. Its constructor and destructor print to the 
terminal, so we can see later when it is actually automatically deleted: 


class Foo 
{ 

public: 

string name; 

Foo(string n) 

: name{move(n)} 

{ cout << "CTOR " << name << '\n'; } 

~Foo() { cout << "DTOR " << name << '\n'; } 

} ; 


3. In order to see what limitations a function has that accepts unique 
pointers as arguments, we just implement one. It processes a Foo item 
by printing its name. Note that while unique pointers are smart, 
overhead-free, and comfortably safe, they can still be null. This means 
that we still have to check them before we dereference them: 

void process^item(unique_ptr<Foo> p) 

{ 

if (!p) { return; } 

cout << "Processing " << p->name << '\n 1 ; 

} 

/v 

4. In the main function, we will open another scope, create twoA Foo 

objects on the heap, and manage both with unique pointers. We create 

/\ 

the first one explicitly on the heap using theA newA operator and then 

/V 

put it into the constructor of the unique_ptr<Foo> variable,A pi. We 



/V 

create the unique pointer,A p2, by calling make_unique<Foo> with the 

arguments we would otherwise directly give the constructor of Foo. This 

/\ 

is the more elegant wayA because we can use auto type deduction and 
the first time we can access the object, it is already managed by 

unique ptrl 

int main() 

{ 

{ 

unique_ptr<Foo> pi {new Foo{"foo"}}; 

auto p2 (make_unique<Foo>("bar")); 

} 

5. After we left the scope, both objects are destructed immediately and 
their memory is released to the heap. Let's have a look at the 
processitem function and how to use it with uniqueptr now. If we 
construct a new Foo instance, managed by a unique ptr in the function 
call, then its lifetime is reduced to the scope of the function. When 
process item returns, the object is destroyed: 

process^item(make_unique<Foo>("fool")); 

6. If we want to call processitem with an object that already existed 

/\ 

before the call, then we need to transfer ownership A because that 

/V 

function takes a unique ptr by value, which meansA that calling it 
would lead to a copy. But unique ptr cannot be copied, it can only be 
moved. Let's create two new Foo objects and move one into 

process item. By looking at the terminal output later, we will see that 

/\ 

foo2 is destroyed when process item returnsA because we transferred 
ownership to it. foo3 will continue living until the main function returns: 

auto pi (make_unique<Foo>("foo2")); 
auto p2 (make_unique<Foo>("foo3")); 

process^item(move(pi)); 

cout << "End of main()\n"; 

} 

7. Let's compile and run the program. At first, we see the constructor and 
destructor calls of foo and bar. They are indeed destroyed just after the 



program leaves the additional scope. Note that the objects are destroyed 
in the opposite order of their creation. The next constructor line comes 
from fool, which is the item we created during the process item call. It 
is indeed destroyed immediately after the function call. Then we created 
foo2 and foo3. foo2 is destroyed immediately after the process item 
call where we transferred the ownership. The other item,A foo3, in 
comparison, is destroyed after the last code line in the main function: 

$ ./unique_ptr 
CTOR foo 
CTOR bar 
DTOR bar 
DTOR foo 
CTOR fool 
Processing fool 
DTOR fool 
CTOR foo2 
CTOR foo3 
Processing foo2 
DTOR foo2 
End of main() 

DTOR foo3 


How it works... 

Handling heap objects with std: :unique_ptr is really simple. After we 
initialized a unique pointer to hold a pointer to some object, there is no way 
we can accidentally forget about deleting it on some code path. 

If we assign some new pointer to a unique pointer, then it will always first 
delete the old object it pointed to and then store the new pointer. On a unique 

/V 

pointer variable,A x, we can also call x. reset () to just delete the object it 
points to immediately without assigning a new pointer. Another equivalent 

/V 

alternative to reassigning viaA X = new_pointer is X. reset (new_pointer) . 


Note 



There is indeed one single way to release an object from the 
management of unique ptr without deleting it. The release 
function does that, but using this function is not advisable in most 
situations. 

Since pointers need to be checked before they are actually dereferenced, they 
overload the right operators in a way that enables them to mimic 

/V 

rawA pointers. Conditionals like if (p) {...} and if (p != nuiiptr) 

/V /V 

{...} perform the same wayA as we would checkA a raw pointer. 

Dereferencing a unique pointer can be done via the get () function, which 
returns a raw pointer to the object that can be dereferenced, or directly via 
operator*, which again mimics raw pointers. 

One important characteristic of unique ptr is that its instances cannot be 
copiedA but can be moved from one unique ptr variable to the other. This is 
why we had to move an existing unique pointer into the process item 
function. If we were able to copy a unique pointer, then this would mean that 
the object being pointed to is owned by two unique pointers, although this 
contradicts the design of a unique pointer that is the onlyowner (and later the 
"deleter") of the underlying object. 

Note 

/\ 

Since there are data structures, such asA unique ptr and 
shared ptr, there is rarely any reason to create heap objects 
directly with new and delete them manually. Use such classes 
wherever you can! Especially unique ptr imposes no overhead 
at runtime. 



Automatically handling shared heap 
memory with std::shared_ptr 


In the last recipe, we learned how to use unique ptr. This is an enormously 

/V 

useful and important classA because it helps us manage dynamically 
allocated objects. However, it can only handle single ownership. It is not 
possible to let multiple objects own the same dynamically allocated object 
because, then, it would be unclear who has to delete it later. 

/v 

The pointer type,A shared ptr, was designed for specifically this case. 
Shared pointers can be copied arbitrarily often. An internal reference 
counting mechanism tracks how many objects are still maintaining a pointer 
to the payload object. Only the last shared pointer that goes out of scope will 
call delete on the payload object. This way, we can be sure that we do not 

/V 

get memory leaks A because objects are deleted automatically after use. At 
the same time, we can be sure that they are not deleted too early, or too often 
(every created object must only be deleted once). 

/\ 

In this recipe, youA will learn how to use shared ptr to automatically 
manage dynamic objects that are shared between multiple owners and see 

/V 

what's different when comparing it withA unique ptr: 

How to do it... 

We are going to write a program that is similar to the program we wrote in 

/\ 

the unique ptr recipe in order to get insights into the usageA and principles 

of shared_ptr: 

1. At first, we just include the necessary headers and declare that we use 

/V 

theA std namespace by default: 


#include <iostream> 
#include <memory> 




using namespace std; 


2. Then we define a little helper class, which helps us see when instances 
of it are actually created and destroyed. We will manage instances of it 

with shared_ptr: 

class Foo 
{ 

public: 

string name; 

Foo(string n) 

: name{move(n)} 

{ cout << "CTOR " << name << '\n 1 ; } 

~Foo() { cout << "DTOR " << name << '\n 1 ; } 

} ; 

3. Next, we implement a function that takes a shared pointer to a Foo 
instance by value. Accepting shared pointers as arguments by value is 

/V 

more interesting than accepting them by referenceA because in this case, 
they need to be copied, which changes their internal reference counter, 
as we will see: 

void f(shared_ptr<Foo> sp) 

{ 

cout << "f: use counter at " 

<< sp.use_count() << '\n'; 

} 

/\ /v 

4. In the main function, we declare anA empty shared pointer. ByA default 

/V 

constructing it, it is A effectively a null pointer: 

int main() 

{ 

shared_ptr<Foo> fa; 

5. Next, we open another scope and instantiate two Foo objects. We create 

/\ /V 

the first one using theA newA operator and then feed it into the 
constructor of a new shared ptr. Then we create the second instance 
using make_shared<Foo>, which creates a Foo instance from the 
parameters we provide. This is the more elegant methodA because we 



can use auto type deduction and the object is already managed when we 
have the chance to access it for the first time. This is very similar to the 
unique ptr recipe at this point: 


{ 

cout << "Inner scope begin\n"; 

shared_ptr<Foo> fl {new Foo{"foo"}}; 

auto f2 (make_shared<Foo>("bar")); 

6. Since shared pointers can be shared, they need to track how many 

/v 

parties share them. This is done with an internal reference counterA or 
use counter. We can print its value using use count. The value 

/\ /V 

isA exactly 1 at this pointA because we did not copy it yet. We can copy 
f 1 to fa, which increases the use counter to 2 . 

cout << "fl's use counter at " << f1.use_count() << 
fa = f1; 

cout << "fl's use counter at " << f1.use_count() << 

7. While we're leaving the scope, the shared pointers f 1 and f 2 are 
destroyed. TheA f lA variable's reference counter is decremented to 1 
again, making fa the only owner of the Foo instance. While f 2 is 
destroyed, its reference counter is decremented to 0 . In this case, 

/V /V 

the A sharedptrA pointer's destructor will call delete on this object, 
which disposes of it: 

} 

cout << "Back to outer scope\n"; 
cout << fa.use_count() << 1 \n 1 ; 

/v 

8. Now, let's call theA f function with our shared pointer in two different 

_ /V /\ 

ways. At first, we call it naively by copying fa. TheA f A function will 

/V 

then print that the reference counter has the valueA 2 . In the second call 
to f , we move the pointer into the function. This makes f the only owner 
of the object: 

cout << "first f() call\n"; 
f(fa); 

cout << "second f() call\n"; 



f(move(fa)); 


/\ 

9. After f A is returned, the Foo instance is destroyed immediately because 

/V _ 

we do not have ownership ofA it any longer. Therefore, all the objects 
are already destroyed when the main function returns: 

cout << "end of main()\n"; 

} 

10. Compiling and running the program yields the following output. In the 

/V 

beginning, we see "foo" and "bar" created. After we copiedA fi 
(which points to "foo"), its reference counter was incremented to 2 . 
While leaving the scope, "bar" is destroyed because the shared pointer 

/V /V 

to it being the A subject ofA destruction is the only owner. The single 1 

/V 

in the output is the reference count of fa, whichA is now the only owner 
of "foo". Afterward, we called function f twice. On the first call, we 
copied fa into it, which gave it a reference counter of 2 again. On the 
second call, we moved it into f , which did not alter its reference counter. 
Moreover, because f is the only owner of "foo" at this point, the object 
is destroyed immediately after f leaves the scope. This way, no other 

/V 

heap objects are destroyed after the last print line A in main: 

$ . /shared_ptr 
Inner scope begin 
CTOR foo 
CTOR bar 

f1' s use counter at 1 
f1's use counter at 2 
DTOR bar 

Back to outer scope 
1 

first f() call 

f: use counter at 2 

second f() call 

f: use counter at 1 

DTOR foo 

end of main() 


How it works... 


When constructing and deleting objects, shared ptr works basically like 



unique ptr. Constructing shared pointers works similarly as creating unique 

/V 

pointers (although there is a function make sharedA that creates shared 

/V 

objects as a pendant to unique ptrA pointer's make unique function). 

, /v 

The major difference fromA unique ptr is that we can copy 
theA shared ptr instances because shared pointers maintain a so-called 
control block together with the object they manage. The control block 
contains a pointer to the payload object and a reference counter or use 

/\ /V 

counter. If there are nA number ofA sharedptr instances pointing to the 

/V 

object, then the use counter also has the valueA n. Whenever a shared ptr 
instance is destructed, then its destructor decrements this internal use counter. 
The last shared pointer to such an object will hit the condition that it 
decrements the use counter to o during its destruction. This is, then, the 
shared pointer instance, which calls the delete operator on the payload 
object! This way, we can't possibly suffer from memory leaks because the 
object's use count is automatically tracked. 

To illustrate this a bit more, let's have a look at the following diagram: 



1 .) 


2 .) 


control block 



3 .) 4 .) 



/V 

In step 1, we have twoA shared ptr instances managing an object of type 
Foo. The use counter is at value 2. Then, shared_ptr2 is destroyed, which 

decrements the use counter to l. The foo instance is not destroyed 

/\ 

yet A because there is still the other shared pointer. In step 3, the last shared 
pointer is destroyed too. This leads to the use counter being decremented to o. 
Step 4 happens immediately after step 3. Both the control block and the 
instance of Foo are destroyed and their memory is released to the heap. 




















Equipped with shared ptr and unique ptr, we can automatically deal with 
most dynamically allocated objects without having to worry about memory 
leaks any longer. There is, however, one important caveat to consider— 
imagine we have two objects on the heap that contain shared pointers to each 
other, and some other shared pointer points to one of them from somewhere 
else. If that external shared pointer goes out of scope, then both objects still 
have the use counters with nonzero values because they reference each other. 
This leads to a memory leak. Shared pointers should not be used in this 

/V 

caseA because such cyclic reference chains prevent the use counter of such 
objects to ever reach o. 

There’s more... 

Look at the following code. What if you are told that it contains a potential 
memory leak ? 

void function(shared_ptr<A>, shared_ptr<B>, int); 

// "function" is defined somewhere else 

// ...somewhere later in the code: 

function(new A{}, new B{}, other_function()); 

"Where is the memory leak?", one might ask, since the newly allocated 
objects a and b are immediately fed into shared ptr types, and then we are 
safe from memory leaks. 

Yes, it is true that we are safe from memory leaks as soon as the pointers are 
captured in the shared ptr instances. The problem is a bit fiddly to grasp. 

/V 

When we call a function,A f (x () , y (), z ()), the compiler needs to 
assemble code that calls x (), y () , and z () first so that it can forward their 
return values to f . What gets us very bad in combination with the example 
from before is that the compiler can execute these function calls to x, y, and z 
in any order. 

Looking back at the example, what happens if the compiler decides to 
structure the code in a way where at first new a{ } is called, then 
other function (), and then new b {} is called, before the results of these 



functions are finally fed into function? If other_function () thfOWS an 
exception, we get a memoiy leak because we still have an unmanaged 

/V 

object,A a, on the heap because we just allocated it but did not have a chance 

/\ 

to handA it to the management of shared ptr. No matter how we catch the 

/V 

exception,A the handle to the object is gone and we cannot delete it! 

There are two easy ways to circumvent this problem: 

// l.) 

function(make_shared<A>(), make_shared<B>(), other_function()); 

// 2 .) 

shared_ptr<A> ap {new A{}}; 
shared_ptr<B> bp {new B{}}; 
function(ap, bp, other_function()); 

This way, the objects are already managed by shared ptr, no matter who 
throws what exception afterward. 



Dealing withA weak pointers toA shared 
obj ects 


In the recipe about shared ptr, we learned how useful and easy to use 
shared pointers are. Together with unique ptr, they pose an invaluable 
improvement for code that needs to manage dynamically allocated objects. 

/V 

Whenever we copyA shared ptr, we increment its internal reference 
counter. As long as we hold our shared pointer copy, the object being pointed 

/V 

toA will not be deleted. But what if we want some kind of weak pointer, 

/V 

whichA enables us to get at the object as long as it exists but does not prevent 
its destruction? And how do we determine if the object still exists, then? 

In such situations, weak ptr is our companion. It is a little bit more 
complicated to use than unique ptr and shared ptr, but after following this 
recipe, we will be ready to use it. 

How to do it... 

We will implement a program that maintains objects with sharedptr 

/V /V 

instances, and then, we mix inA weak ptrA to see how this changes the 

/V /V 

behavior A of smart pointer A memory handling: 

1. At first, we include the necessary headers and declare that we use the 
std namespace by default: 

#include <iostream> 

#include <iomanip> 

#include <memory> 

usinq namespace std; 


2. Next, we implement a class that prints a message in its destructor 
implementation. This way, we can simply check when an item is 




actually destroyed later in the program output: 


struct Foo { 
int value; 

Foo(int i) : value{i} {} 

~Foo() { cout << "DTOR Foo " << value << '\n'; } 


3. Let's also implement a function that prints information about a weak 
pointer, so we can print a weak pointer's state at different points of our 
program. The expired function of weakptr tells usA if the object it 
points to still really exists, because holding a weak pointer to an object 

_ /\ /V 

does not prolong its lifetime! TheA use countA counter tells us how 
many shared ptr instances are currently pointing to the object in 
question: 


void weak_ptr_info(const weak_ptr<Foo> &p) 

{ 

cout << "-" << boolalpha 

<< "\nexpired: " << p.expired() 

<< "\nuse_count: " << p.use_count() 

<< "\ncontent: 

4. If we want to access the actual object, we need to call the lock function. 
It returns us a shared pointer to the object. In case the object does not 
exist any longer, the shared pointer we got from it is effectively a null 
pointer. We need to check that, and then we can access it: 

if (const auto sp (p.lockO); sp) { 
cout << sp->value << 1 \n 1 ; 

} else { 

cout << "<null>\n"; 

} 

} 

5. Let's instantiate an empty weak pointer in the main function and print its 
content which is, of course, empty at first: 


int main() 
{ 


weak ptr<Foo> weak foo; 




weak_ptr_info(weak_foo); 


6. In a new scope, we instantiate a new shared pointer with a fresh instance 

/V /V 

of theA FooA class. Then we copy it to the weak pointer. Note that this 
will not increment the reference count of the shared pointer. The 

/V /V 

reference counter is lA because only one A shared pointer owns it: 

{ 

auto shared_foo (make_shared<Foo>(1337)); 
weak foo = shared foo; 


7. Let's call the weak pointer function before we leave the scope and, 

/\ 

again,A after we leave the scope. The Foo instance should be destroyed 
immediately, although a weak pointer points to it: 


weak_ptr_info(weak_foo); 

} 

weak_ptr_info(weak_foo); 

} 

8. Compiling and running the program yields us three times the output of 
the weak ptr inf o function. In the first call, the weak pointer is empty. 
In the second call, it already points to the Foo instance we createdA and 
is able to dereference it after locking it. Before the third call, we leave 
the inner scope, which triggers the destructor of the Foo instance, as we 
expected. Afterward, it is not possible to get at the content of the deleted 
Foo item via the weak pointer any longer, and the weak pointer correctly 
recognizes that it has expired: 

$ ./weak_ptr 


expired: true 
use_count: 0 
content: <null> 


expired: false 
use_count: 1 
content: 1337 
DTOR Foo 1337 


expired: true 
use count: 0 






content: <null> 


How it works... 

Weak pointers provide us a way to point at an object maintained by shared 
pointers without incrementing its use counter. Okay, a raw pointer could do 
the same, but a raw pointer cannot tell us if it is dangling or not. A weak 
pointer can! 

In order to understand how weak pointers as an addition to shared pointers 
work, let's directly jump to an illustrating diagram: 



1 .) 2 .) 





The flow is similar to the diagram in the recipe about shared pointers. In step 
1, we have two shared pointers and a weak pointer pointing to the object of 
type Foo. Although there are three objects pointing to it, only the shared 

pointers manipulate its use counter, which is why it has the valueA 2 . The 

/\ 

weak pointer only manipulates aA weak counter of the control block. In steps 
2 and 3, the shared pointer instances are destroyed, which leads stepwise to a 
use counter of o. In step 4, this results in the foo object being deleted, but the 
control block stays there. The weak pointer still needs the control block in 
order to distinguish if it dangles or not. Only when the lastweak pointer that 






























still points to a control block also goes out of scope, the control block is 
deleted. 


We can also say that a dangling weak pointer has expired. In order to check 

/V 

for this attribute, we can ask weak ptrA pointer's expired method, which 
returns a boolean value. If it is true, then we cannot dereference the weak 
pointer because there is no object to dereference any longer. 

In order to dereference a weak pointer, we need to call lock (). This is safe 

/V 

and convenientA because this function returns us a shared pointer. As long as 
we hold this shared pointer, the object behind it cannot vanish because we 
incremented the use counter by locking it. If the object is deleted, shortly 
before the lock () call, then the shared pointer it returns is effectively a null 
pointer. 



Simplifying resource handling of legacy 
APIs with smart pointers 


Smart pointers (unique ptr, shared ptr, and weak ptr) are extremely 
useful, and it is, in general, safe to say that a programmer should always use 
these instead of allocating and freeing memory manually. 

But what if objects cannot be allocated using the new operator and/or cannot 
be freed again using delete? Many legacy libraries come with their own 
allocation/destruction functions. It seems that this would be a 

/V 

problemA because we learned that smart pointers rely on new and delete. If 
the creation and/or destruction of specific types of objects relies on specific 
factory functions' deleter interfaces, does this prevent us from getting the 
humongous benefits of smart pointers? 

Not at all. In this recipe, we will see that we only need to perform very 
minimal customizations on smart pointers in order to let them follow specific 
procedures for allocation and destruction of specific objects. 

How to do it... 

In this section, we will define a type that cannot be allocated with new 
directly and, also, cannot be released again using delete. As this prevents it 

from being used with smart pointers directly, we perform the necessary 

/\ /\ 

littleA adaptions to instances ofA unique ptr and smart ptr: 

1. As always, we first include the necessary headers and declare that we 

/V 

use the A std namespace by default: 

#include <iostream> 

#include <memory> 

#include <string> 


using namespace std; 



2. Next, we declare a class that has its constructor and destructor declared 
private. This way, we simulate the problem that we have to access 
specific functions that create and destroy instances of it: 

class Foo 

{ 

string name; 

Foo(string n) 

: namefn} 

{ cout << "CTOR " << name << '\n 1 ; } 

~Foo() { cout << "DTOR " << name << '\n 1 ;} 

3. The static methods,A create foo and destroy foo, then create and 
destroy theA foo instances. They work with raw pointers. This simulates 
the situation of a legacy C API, which prevents us from using them with 

/V 

normal shared ptrA pointers directly: 

public: 

static Foo* create_foo(string s) { 
return new Foo{move(s)}; 

} 

static void destroy_foo(Foo *p) { delete p; } 

}; 

/V 

4. Now, let's make such objects manageable byA shared ptr. We can, of 

course, put the pointer we get from createfoo into the constructor of a 

/\ 

shared pointer. Only the destruction is trickyA because the default 
deleter of shared ptr would do it wrong. The trick is that we can give 

_ /V 

sharedptr a custom deleter. The function signature that A a deleter 
function or callable object needs to have is already the same as that of 

/V 

theA destroy f oo function. If the function we need to call for 
destroying the object is more complicated, we can simply wrap it into a 
lambda expression: 

static shared_ptr<Foo> make_shared_foo(string s) 

{ 

return {Foo::create_foo(move(s)), Foo::destroy_foo}; 

} 



5. Note that make_shared_foo returns a usual shared_ptr<Foo> 

/v 

instance A because giving it a custom deleter did not change its type. 

^ /v 

This isA because shared ptr uses virtual function calls to hide such 
details. Unique pointers do not impose any overhead, which makes the 
same trick unfeasible for them. Here, we need to change the type of the 
unique ptr. As a second template parameter, we give it void (*) 

(Foo* ), which is exactly the type of pointer to the 
function,A destroy_foo: 

static unique_ptr<Foo, void (*)(Foo*)> make_unique_foo(str 
{ 

return {Foo::create_foo(move(s)), Foo::destroy_foo}; 

} 

6. In the main function, we just instantiate both a shared pointer and a 
unique pointer instance. In the program output, we will see if they are 
really, correctly, and automatically destroyed: 


int main() 

{ 

auto ps (make_shared_foo("shared Foo instance")); 
auto pu (make_unique_foo("unique Foo instance")); 

} 

7. Compiling and running the program yields the following output, which 
is luckily just what we expected: 

$ ./leqacy_shared_ptr 
CTOR shared Foo instance 
CTOR unique Foo instance 
DTOR unique Foo instance 
DTOR shared Foo instance 


How it works... 

Usually, unique ptr and shared ptr just call delete on their internal 
pointers, whenever they ought to destroy the object they maintain. In this 
section, we constructed a class which can neither be allocated the C++ way 

/V 

using x = new Foo {123 } A nor can it be destructed with delete x directly. 



TheA Foo:: create f oo function just returns a plain raw pointer to a newly 

/\ 

constructed foo instance, so this causes no further problemsA because smart 
pointers work with raw pointers anyway. 

The problem we had to deal with is that we need to teach unique ptr and 
sharedptr how to destruct an objectA if the default way is not the right one. 

In that regard, both the smart pointer types differ a little bit. In order to define 
a custom deleter for unique ptr, we have to alter its type. Because the type 
signature of the Foo deleter is void Foo:: destroyf oo (Foo* );, the type of 

/v 

the unique ptr maintaining a Foo instance must beA unique_ptr<Foo, void 
(*) (Foo*) >. Now, it can hold a function pointer to destroy foo, which we 
provide it as a second constructor parameter in our make unique f oo 
function. 

If giving unique ptr a custom deleter function forces us to change its type, 
why were we able to do the same with shared_ptr without changing its type? 
The only thing we had to do there was giving shared ptr a second 
constructor parameter, and that's it. Why can't it be as easy for 

/V /V 

unique_ptrA as it is forA shared_ptr? 

The reason why it is so simple to just provide shared ptr some kind of 
callable deleter object without altering the shared pointer's type lies in the 
nature of shared pointers, which maintain a control block. The control block 
of shared pointers is an object with virtual functions. This means that the 

/V 

control block of a standard shared pointer compared withA the type of a 
control block of a shared pointer with a custom deleter is different ! When we 
want a unique pointer to use a custom deleter, then this changes the type of 
the unique pointer. When we want a shared pointer to use a custom deleter, 
then this changes the type of the internal control block, which is invisible to 
us because this difference is hidden behind a virtual function interface. 

It would be possible to do the same trick with unique pointers, but then, this 
would imply a certain runtime overhead on them. This is not what we want 
because unique pointers promise to be completelyA overhead free at runtime. 



Sharing different member valuesA of the 
same object 


Let's imagine we are maintaining a shared pointer to some complex, 
composed, and dynamically allocated object. Then, we want to start a new 
thread that does some time-consuming work on a member of this complex 
object. If we want to release this shared pointer now, the object will be 
deleted while the other thread is still accessing it. If we don't want to give the 
thread object the pointer to the whole complex object because that would 
mess with our nice interface, or for other reasons, does this mean that we 
have to do manual memory management now? 

No. It is possible to use shared pointers that on one hand, point to a member 

of a large shared object, while on the other hand, perform automatic memory 

/\ 

management for the entire A initial object. 

In this example, we will create such a scenario (without threads to keep it 
simple) in order to get a feeling for this handy feature of shared ptr. 

How to do it... 

We are going to define a structure that is composed of multiple members. 
Then, we allocate an instance of this structure on the heap that is maintained 
by a shared pointer. From this shared pointer, we obtain more shared pointers 
that do not point to the actual object but to its members: 

1. We include the necessary headers first and then declare that we use 

/V 

theA std namespace by default: 

#include <iostream> 

#include <memory> 

#include <string> 


using namespace std; 





2. Then we define a class that has different members. We will let shared 
pointers point to the individual members. In order to be able to see when 
the class is created and destroyed, we let its constructor and destructor 
print messages: 


struct person { 
string name; 
size_t age; 

person(string n, size_t a) 

: name{move(n)}, age{a} 

{ cout << "CTOR " << name << '\n 1 ; } 

-person () { cout << "DTOR " << name << '\n'; } 

}; 

/V /V 

3. Let's define shared pointers thatA have the right types to point toA the 
name and age member variables of a person class instance: 


int main() 

{ 

shared_ptr<string> shared_name; 
shared_ptr<size_t> shared_age; 


4. Next, we enter a new scope, create such a person object, and let a shared 
pointer manage it: 

{ 

auto sperson (make_shared<person>("John Doe", 30)); 


5. Then, we let the first two shared pointers point to its name and age 
members. The trick is that we use a specific constructor of shared ptr, 
which accepts a shared pointer and a pointer to a member of the shared 
object. This way, we can manage the object while not pointing at the 
object itself! 


shared_name = shared_ptr<string>(sperson, Ssperson- 
shared_age = shared_ptr<size_t>(sperson, Ssperson- 

} 

6. After leaving the scope, we print the person's name and age values. This 
is only legalA if the object is still allocated: 



} 


cout << "name: 
<< "\nage: 


If 


<< *shared_name 
<< *shared_age << '\n'; 


7. Compiling and running the program yields the following output. From 

the destructor message, we see that the object is indeed still alive and 

/\ 

allocatedA when we access the person's name and age values via the 
member pointers! 

$ ./shared_members 
CTOR John Doe 
name: John Doe 
age:30 

DTOR John Doe 


How it works... 

In this section, we first created a shared pointer that manages a dynamically 
allocated person object. Then we made two other smart pointers point to the 
person object, but they both did not directly point to the person object itself 

/v /\ 

butA instead to its members,A name and age. 

To summarize what kind of scenario we just created, let's have a look at the 
following diagram: 




Note that shared ptri points to the person object directly, while 
shared name and shared age point to the name and the age members of the 
same object. Apparently, they still manage the object's entire lifetime. This is 
possible because the internal control block pointers still point to the same 
control block, no matter what sub-object the individual shared pointers point 
to. 

In this scenario, the use count of the control block is 3. This way, the person 

/\ 

object is not destroyed when shared ptri is destroyedA because the other 









shared pointers still own the object. 


When creating such shared pointer instances that point to members of the 
shared object, the syntax looks a bit strange. In order to obtain a 
shared_ptr<string> that points to the name member of a shared person, we 
need to write the following: 


auto sperson (make_shared<person>("John Doe", 30)); 

auto sname (shared_ptr<string>(sperson, &sperson->name)); 

In order to get a specific pointer to a member of a shared object, we 
instantiate a shared pointer with a type specialization of the member we want 
to access. This is why we write shared_ptr<string>. Then, in the 
constructor, we first provide the original shared pointer that maintains the 
person object and, as a second argument, the address of the object the new 
shared pointer willA use when we dereference it. 



Generating random numbers and choosing 
the right random number engine 


In order to get random numbers for whatever purpose, C++ programmers 
usually basically used the rand () function of the C library before C++11. 

/V 

Since C++11, there has beenA a whole suite of random number generators 

/V 

thatA serve different purposes and have different characteristics. 

These generators are not completely self-explanatory, so we will have a look 
at all of them in this recipe. In the end, we will see in what ways they differ, 
how to choose the right one, and that we will most probably never use all of 
them. 


How to do it... 


We will implement a procedureA that prints a nice illustrating histogram of 
the numbers a random generator produces. Then, we will run all STL random 
number generator engines through this procedure and learn from the results. 
This program contains many repetitive lines, so it might be advantageous to 
just copy the source code from the code repository accompanying this book 
on the Internet instead of typing all the repetitive code manually. 

/V 

1. At first, we include all the necessary headersA and then declare that we 

/V 

use the A std namespace by default: 

#include <iostream> 

#include <string> 

#include <vector> 

#include <random> 

#include <iomanip> 

#include <limits> 

#include <cstdlib> 

#include <algorithm> 


using namespace std; 




/v 

2. Then we implement a helper function, which helps us maintainA and 
print some statistics for each type of random number engine. It accepts 
two parameters: the number of partitions and the number of samples. 
We will see immediately what these are for. The type of random 
generator is defined via the template parameter rd. The first thing we do 
in this function is define an alias type for the resulting numeric type of 
the numbers the generator returns. We also make sure that we have at 
least 10 partitions: 


template <typename RD> 

void histogram(size_t partitions, size_t samples) 

{ 

using rand_t = typename RD::result_type; 
partitions = max<size_t>(partitions, 10); 

3. Next, we instantiate an actual generator instance of type rd. Then, we 
define a divisor variable called div. All random number engines emit 
random numbers within the range from o to rd :: max (). The function 

/V 

argument,A partitions, allows the caller to choose by how many 
partitions we divide every random number range. By dividing the largest 
possible value by the number of partitions, we know how large every 
partition is: 

RD rd; 

rand_t div ((double(RD::max()) + 1) / partitions); 

4. Next, we instantiate a vector of counter variables. It is exactly as large as 
the number of partitions we have. Then, we get as many random values 
out of the random engine as the variable samples says. The 

/V 

expression,A rd (), gets a random number from the generator and shifts 
its internal state to prepare it for returning the next random number. By 
dividing every random number by div, we get the partition number it 
falls into and can increment the right counter in the vector of counters: 

vector<size_t> v (partitions); 

for (size_t i {0}; i < samples; ++i) { 

++v[rd() / div]; 

} 


5. Now we have a nice coarse-grained histogram of sample values. In order 



to print it, we need to know a little bit more about its actual counter 
values. Let's extract its largest value using the max eiement algorithm. 
We then divide this largest counter value by 100 . This way, we can 
divide all the counter values by max_div and print a lot of stars on the 

/V 

terminal without exceeding theA width of 100 . If the largest counter 
contains a number less than 100 , because we did not use so many 
samples, we use max in order to get a minimal divisor of 1 : 

rand_t max_elm (*max_element(begin(v), end(v))); 
rand_t max_div (max(max_elm / 100, rand_t(l))); 

6. Let's now print the histogram to the terminal. Every partition gets its 
own line on the terminal. By dividing its counter value by max_div and 
print so many asterisk symbols we get histogram lines that fit into 
the terminal: 


} 


for (size_t i {0}; i < partitions; ++i) { 

cout << setw(2) << i << " 

<< string(v[i] / max_div, 1 *') << '\n 1 ; 

} 


7. Okay, that's it. Now to the main program. We let the user define how 
many partitions and samples should be used: 

int main(int argc, char **argv) 

{ 

if (argc != 3) { 

cout << "Usage: " << argv[0] 

<< " <partitions> <samples>\n"; 
return 1; 

} 


8. We then read those variables from the command line. Of course, the 
command line consists of strings, which we can convert to numbers 

/\ /V 

using std:: stouiiA (stouiiA is an abbreviation for string to unsigned 
long long): 


size_t partitions {stoull(argv[1]) } ; 
size_t samples {stoull(argv[2])}; 


9. Now we call our histogram helper function on every random number 



engine the STL provides. This makes this recipe very long and 
repetitive. Better copy the example from the Internet. The output of this 
program is really interesting to look at. We start with random device. 
This device tries to distribute the randomness equally over all the 
possible values: 

cout << "random_device" << '\n'; 
histogram<random_device>(partitions, samples); 

10. The next random engine we try is defauit random engine. What kind 
of engine this type refers to is implementation-specific. It can be any of 
the following random engines: 

cout << "ndefault_random_engine" << ' \n 1 ; 
histograrrKdefault_random_engine> (partitions, samples) ; 


11. Then we try it on all the other engines: 

cout << "nminstd_randO" << ' \n' ; 
histogram<minstd_randO>(partitions, samples); 
cout << "nminstd_rand" << '\n'; 
histogram<minstd_rand>(partitions, samples); 

cout << "nmtl9937" << '\n'; 
histograrrKmt 19937> (partitions, samples) ; 
cout << "nmtl9937_64" << '\n'; 
histograrrKmt 19937_64> (partitions, samples) ; 

cout << "nranlux24_base" << '\n'; 
histogram<ranlux24_base>(partitions, samples); 
cout << "nranlux48_base" << '\n'; 
histogram<ranlux48_base>(partitions, samples); 

cout << "nranlux24" << '\n'; 
histogram<ranlux24>(partitions, samples); 
cout << "nranlux48" << '\n'; 
histogram<ranlux48>(partitions, samples); 

cout << "nknuth_b" << '\n'; 
histogram<knuth_b>(partitions, samples); 


12. Compiling and running the program yields interesting results. We will 
see a long list of output, and we'll see that all the random engines have 



different characteristics. Let's first run the program with 10 partitions 
and only 1000 samples: 


$ ./random_generator 10 1000 
random_devi.ce 

************************************************************************************** 

2 . ******************************************************************************************************* 

3. ************************************************************************************************ 

4. *********************************************************************************************************** 

3. ************************************************************************************************************************* 

0 . ************************************************************************************************************ 

7. ******************************************************************************* 

g- ****************************************************************************************************** 

9 ; *********************************************************************************************** 

default_random_engine 

0 . ************************************************************************************************ 
************************************************************************************************* 

2 • ************************************************************************************************* 

3. ********************************************************************************************* 

4. ********************************************************************************************************************* 

5 ; *************************************************************************************************************** 

0 . ************************************************************************************************************** 

7. ********************************************************************************************** 

g. ************************************************************************************************ 

9. ***************************************************************************************** 

minstd_rand0 

0. ************************************************************************************************ 

************************************************************************************************* 

2 : ************************************************************************************************* 

3. ********************************************************************************************* 

4. ********************************************************************************************************************* 

5 ; *************************************************************************************************************** 

0 . ************************************************************************************************************** 

7. ********************************************************************************************** 

g. ************************************************************************************************ 

9. ***************************************************************************************** 




13. Then, we run the same program again. This time it is still 10 partitions 
but 1 , 000,000 samples. It becomes very obvious that the histograms 
look much cleaner, when we take more samples from them. This is an 
important observation: 



$ ./random_generator 10 1000000 
random_device 

0 . *************************************************************************************************** 

1 • *************************************************************************************************** 

2 ; *************************************************************************************************** 

3. *************************************************************************************************** 

4 ; **************************************************************************************************** 
5. *************************************************************************************************** 

0 . *************************************************************************************************** 
7. *************************************************************************************************** 

g. *************************************************************************************************** 

g. *************************************************************************************************** 
default_random_engine 

q. *************************************************************************************************** 

]_ • *************************************************************************************************** 
2 » *************************************************************************************************** 

3• *************************************************************************************************** 

4. *************************************************************************************************** 

5. **************************************************************************************************** 

g. *************************************************************************************************** 

7. *************************************************************************************************** 

g. *************************************************************************************************** 

g• *************************************************************************************************** 
minstd_rand0 

0. *************************************************************************************************** 
*************************************************************************************************** 

2 ; *************************************************************************************************** 

3. *************************************************************************************************** 

4. *************************************************************************************************** 

5• **************************************************************************************************** 
0 . *************************************************************************************************** 
7. *************************************************************************************************** 
g. *************************************************************************************************** 

g. *************************************************************************************************** 


How it works... 

In general, any random number generator needs to be instantiated as an object 
before use. The resulting object can be called like a function without 
parameters because it overloads operator (). Every call will then lead to a 
new random number. It is that simple. 

In this section, we wrote a program that is much more complex than that in 

/\ 

order to get a bit more informationA about random number generators. Please 
play around with the resulting program by launching it with different 
command-line arguments and realize the following facts: 

• The more samples we take, the more equal our partition counters appear. 

• The inequality of the partition counters wildly differs between individual 
engines. 

• For a large number of samples, it becomes apparent that the 



/V 

performance of the individualA random engines differs. 

• Run the program with a low amount of samples multiple times. The 

/V 

distribution patterns look the same A all the time—the random engines 
produce the same random number sequences repeatedly, which means 
they are not random at all. Such engines are called 

/V 

deterministic A because their random numbers can be predicted. The 
Only exception is std: : random_device. 

As we can see, there are a few characteristics to consider. For most standard 
applications, std: :defauit_random_engine will be completely sufficient. 
Experts of cryptography or similarly security-sensitive topics will choose 
wisely between the engines they use, but for us average programmers, this is 
not too important when we write apps with some randomness. 

/V 

We should carry home the following threeA facts from this recipe: 

1. Usually, std: : def ault_random_engine is a good default choice for the 
average application. 

2. If we really need non-deterministic random numbers, 
std: : random_device provides US Such. 

3. We can feed the constructor of any random engine with a real random 
number from std:: random device (or maybe a timestamp from the 
system clock), in order to make it produce different random numbers 
each time. This is called seeding. 

Note 

Note that std:: random_devicecai7 possibly fall back to one of 

/V 

the deterministic enginesA if the library has no support for 
nondeterministic random engines. 



/V 

GeneratingA random numbers and letting 
the STL shape specific distributions 


In the last recipe, we learned some bits about the STL random number 
engines. Generating random numbers this or the other way is often only half 
of the work. 

Another question is, what do we need those numbers for? Are we 
programmatically "flipping a coin"? People used to do this using rand () % 2 , 

/V 

which results in values of 0 and 1 that canA then be mapped to head or tail. 
Fair enough; we do not need a library for that (although randomness experts 
know that just using the lowest few bits of a random number does not always 
lead to high-quality random numbers). 

What if we want to model a die? Then, we could surely write ( rand 0 % 6) 

+ 1 , in order to represent the result after rolling the die. There is still no 
pressing library needed for such simple tasks. 

What if we want to model something that happens with an exact probability 
of 66%? Okay, then we can come up with a formula like bool yes no = 

/V 

(rand () % ioo > 66) . (Oh wait, should it be >=, or is >A correct?) 

Apart from that, how do we model an unfair die whose sides do not all have 
the same probability? Or how do we model more complex distributions? 

Such problems can quickly evolve to scientific tasks. In order to concentrate 

/V 

on our primary problems,A let's have a look at what the STL already provides 
in order to help us. 

The STL contains more than a dozen distribution algorithms that can shape 
random numbers for specific needs. In this recipe, we are going to have a 
very brief look at all of them, and a closer look at the most generally useful 


ones. 




How to do it... 


We are going to generate random numbers, shape them, and print their 
distribution patterns to the terminal. This way, we can get to know all of them 
and understand the most important ones, which is useful if we ever need to 
model something specific with randomness in mind: 

1. At first, we include all the needed headers and declare that we use the 

/V 

stdA namespace: 

#include <iostream> 

#include <iomanip> 

#include <random> 

#include <map> 

#include <string> 

#include <algorithm> 

using namespace std; 


2. For every distribution the STL provides, we will print a histogram in 

/V 

order to see its characteristics A because every distribution looks very 
special. It acceptsA a distribution as an argument and the number of 

/V _ 

samples thatA shall be taken from it. Then, we instantiate the default 

A _ 

random engine A and a map. The map maps from the values we obtained 

/V 

from the distribution to counters that countA how often which value 
occurred. The reason for why we always instantiate a random engine is 

/\ /V 

that all distributions are just used as A a shaping function A for random 
numbers that still need to be generated by a random engine: 


template ctypename T> 

void print_distro(T distro, size_t samples) 

{ 

default_random_engine e; 
map<int, size_t> m; 

/v /\ 

3. We take as many samples as theA samples A variable says and feed the 
map counters with them. This way, we get a nice histogram. While 
calling e () alone would get us a raw random number from the random 
engine, distro (e) shapes the random numbers through the distribution 
object. 



for (size_t i {0}; i < samples; ++i) { 

m[distro(e)] += 1; 

} 

4. In order to get a terminal output that fits into the terminal window, we 
need to know the largest counter value. The max eiement function helps 
us in finding the largest value by comparing all the associated counters 
in the map and returning us an iterator to the largest counter node. 
Knowing this value, we can determine by what value we need to divide 
all the counter values in order to fit the output into the terminal window: 


size_t max_elm (max_element(begin(m), end(m), 

[](const auto &a, const auto &b) { 

return a.second < b.second; 

})->second); 

size t max div (max(max elm / 100, size t(l))); 


5. Now, we loop through the map and print a bar of asterisk symbols ' * ' 
for all counters which have a significant size. We drop the others 
because some distribution engines spread the numbers over such large 
domains that it would completely flood our terminal windows: 


for (const auto [randval, count] : m) { 

if (count < max elm / 200) { continue; } 


} 


cout << setw(3) << randval << " : " 

<< string(count / max_div, '*') << '\n 1 ; 


6. In the main function, we check if the user provided us exactly one 
parameter, which tells us how many samples to take from each 
distribution. If the user provided none or multiple parameters, we error 
out. 


int main(int argc, char **argv) 

{ 

if (argc != 2) { 

cout << "Usage: " << argv[0] 
<< " <samples>\n"; 
return 1; 

} 



7. We convert the command-line argument string to a number using 

std::stouli: 

size_t samples {stouli(argv[1])}; 

8. At first, we try the uniform_int_distribution and 

normai distribution. These are the most typical distributions used 

/\ 

where random numbers are needed.A Everyone who ever had stochastic 
as a topic in maths at school will most probably have heard about these 
already. The uniform distribution accepts two values,A denoting the 
lower and the upper bound of the range they shall distribute random 
values over. By choosing o and 9, we will get equally often occurring 

/V /V /V 

values between (including)A oA andA 9. The normal distribution 

/\ 

accepts a mean value and a standard derivation A as arguments: 

cout << "uniform_int_distribution\n"; 

print_distro(uniform_int_distribution<int>{0, 9}, sampl 
cout << "normal_distribution\n"; 

print_distro(normal_distribution<double>{0.0, 2.0}, sam 


9. Another really interesting distribution is 

piecewise_constant_distribution. It accepts two input ranges as 
arguments. The first range contains numbers that denote the limits of 
intervals. By defining it as o, 5 , l o, 3 o, we get one interval that spans 
from o to 4, then, an interval that spans from 5 to 9, and the last interval 
spanning from 10 to 2 9. The other input range defines the weights of the 
input ranges. By setting those weights to o . 2 , 0.3, o. 5, the intervals 
are hit by random numbers with the chances of 20%, 30%, and 50%. 
Within every interval, all the values are hit with equal probability: 

initializer_list<double> intervals {0, 5, 10, 30}; 
initializer_list<double> weights {0.2, 0.3, 0.5}; 
cout << "piecewise_constant_distribution\n"; 
print_distro( 

piecewise_constant_distribution<double>{ 
begin(intervals), end(intervals), 
begin(weights)}, 
samples); 

_ A 

10. The piecewise iinear distributionA is constructed similarly, but its 



weight characteristics work completely differently. For every interval 
boundary point, there is one weight value. In the transition from one 
boundary to the other, the probability is linearly interpolated. We use the 

/V 

same interval list but a different list of weight values. A 

cout << "piecewise_linear_distribution\n"; 
initializer_list<double> weights2 {0, 1, 1, 0}; 
print_distro( 

piecewise_linear_distribution<double>{ 

begin(intervals), end(intervals), begin(weights 
samples); 

11. The Bernoulli distribution is another important distribution because it 

/V 

distributes only yes/no, hit/miss, or head/tail A values with a specific 

/V 

probability. Its output values are onlyA o or l. Another interesting 
distribution, which is useful in many cases, is discrete distribution. 
In our case, we initialize it to the discrete values l, 2 , 4, 8. These 
values are interpreted as weights for the possible output values o to 3: 

cout << "bernoulli_distribution\n"; 

print_distro(std::bernouHindistribution{0.75}, samples 
cout << "discrete_distribution\n"; 

print_distro(discrete_distribution<int>{{1, 2, 4, 8}}, 

12. There are a lot of different other distribution engines. They are very 
special and useful in very specific situations. If you have never heard 

/V 

about them, they may A not be for you. However, since our program will 
produce nice distribution histograms, we will print them all, for curiosity 
reasons: 

cout << "binomial_distribution\n"; 

print_distro(binomial_distribution<int>{10, 0.3}, sampl 
cout << "negative_binomial_distribution\n"; 
print_distro( 

negative_binomial_distribution<int>{10, 0.8}, 
samples); 

cout << "geometric_distribution\n"; 

print_distro(geometric_distribution<int>{0.4}, samples) 
cout << "exponential_distribution\n"; 

print_distro(exponential_distribution<double>{0.4}, sam; 
cout << "gamma_distribution\n"; 

print distro(gamma distribution<double>{1.5, 1.0}, samp 



cout << "weibull_distribution\n"; 

print_distro(weibull_distribution<double>{1.5, 1.0}, sa: 
cout << "extreme_value_distribution\n" ; 
print_distro( 

extreme_value_distribution<double>{0.0, 1.0}, 
samples); 

cout << "lognormal_distribution\n"; 

print_distro(lognormal_distribution<double>{0.5, 0.5}, 
cout << "chi_squared_distribution\n"; 

print_distro(chi_squared_distribution<double>{1.0}, sam 
cout << "cauchy_distribution\n"; 

print_distro(cauchy_distribution<double>{0.0, 0.1}, sam 
cout << "fisher_f_distribution\n"; 

print_distro(fisher_f_distribution<double>{1.0, 1.0}, s 
cout << "student_t_distribution\n"; 

print_distro(student_t_distribution<double>{1.0}, sampl 

} 

13. Compiling and running the program yields the following output. Let's 
first run the program with 1000 samples per distribution: 


$ ./random_distro 1000 
uniform_int_distribution 

0 . ************************************************************************************************ 

2 . ************************************************************************************************* 

2 . ************************************************************************************************* 

3 • ********************************************************************************************* 

4 . ********************************************************************************************************************* 

3 . *************************************************************************************************************** 

g . ************************************************************************************************************** 

7 • ********************************************************************************************** 

g . ************************************************************************************************ 

0 . ***************************************************************************************** 
normal_distribution 
-7 : 

-5 : ** 

_4 . **** 

_3 . ************** 

_2 ; ******************************* 

. **************************************************** 

0 . **************************************************************************************************************************** 
^ . *************************************************** 

2 . ******************************* 

3 . *************** 

4 • ***** 

5 : * 

piecewise_constant_distribution 
0 . ***************************************** 

^ . ************************************* 

2 . ********************************* 

3 . *************************************************** 

4 . ************************************************ 

3 . ***************************************************************** 

0 . ************************************************** 

7 . ************************************************************ 


14. Another run with 1 , 000,000 samples per distribution shows that the 
histograms appear much cleaner and more typical for each distribution. 
But we also see which ones are slow, and which ones are fast, while 



they are being generated: 


$ ./random_distro 1000000 
uniform_int_distribution 

0 . *************************************************************************************************** 

1 . *************************************************************************************************** 

2 . *************************************************************************************************** 

3 . *************************************************************************************************** 

4 . *************************************************************************************************** 

5 • **************************************************************************************************** 

g . *************************************************************************************************** 

2 . *************************************************************************************************** 

g . *************************************************************************************************** 

g . *************************************************************************************************** 
no rmal_distribution 
-5 : * 

_4 . **** 

_3 . *********** 

_2 • ************************ 

_1 • *************************************** 

0 . **************************************************************************************************** 
1 . *************************************** 

2 . ************************ 

0 . *********** 

4 . **** 

5 : * 

piecewise_constant_distribution 

0 . ****************************************************************** 

1 . ****************************************************************** 

2 . ****************************************************************** 

^ . ****************************************************************** 

4 . ****************************************************************** 

5 . *************************************************************************************************** 

g . *************************************************************************************************** 

2 . *************************************************************************************************** 
g . **************************************************************************************************** 

g . **************************************************************************************************** 
10 . ***************************************** 

H . ***************************************** 

12 • ***************************************** 

13 . ***************************************** 

14 . ***************************************** 

15 . ***************************************** 

15 . ***************************************** 

12 ; ****************************************** 
lg . ***************************************** 

10 . ***************************************** 

7ft . ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦_ 


How it works... 

While we usually do not care too much about the random number engine, as 

long it is fast and produces numbers that are as random as possible, the 

/\ 

distribution is something we shouldA choose wisely, depending on the 
problem we like to solve (or create). 

In order to use any distribution, we first need to instantiate a distribution 
object from it. We have seen that different distributions take different 
constructor arguments. In the recipe description, we went a bit too briefly 
over some distribution engines because most of them are too special and/or 



too complex to cover here. But don't worry, they are all documented in detail 
in the C++ STL documentation. 


However, as soon as we have a distribution instantiated, we can call it like a 
function that accepts a random engine object as its only parameter. What 
happens then is that the distribution engine takes a random value from the 
random engine, applies some magic shaping (which completely depends on 
the choice of the distribution engine, of course), and then returns us a shaped 
random value. This leads to completely different histograms, as we 

/V 

saw A after executing the program. 

The most comprehensive way to get to know the different distributions is 
playing around with the program we just wrote. In addition to that, let's 
summarize the most important distributions. For all the distributions that 
occur in our program but not in the following table, please consult the C++ 
STL documentation if you are interested: 


Distribution Description 


This distribution accepts a lower and an upper 
bound value as constructor arguments. It does, 
then, give us random numbers that always fall 
into the interval between (including) those 
. ^ ^ , bounds. The probability for each of the values 

uniform^int_distribution r J 

in this interval is the same, which gives us a 

/V /\ 

histogram with aA flat A. shape. This distribution 

/\ 

is representative ofA rolling a die, for example, 
because each side of the die has the same 
probability to occur. 


The normal distribution, or Gauss distribution, 
occurs practically everywhere in nature. Its STL 
version accepts a mean value and a standard 
derivation value as constructor parameters and 
normaidistribution forms a roof -like shape in the histogram. If we 



compare the body size or IQ of humans or other 
animals, or the grades of students, we will 
realize that these numbers are also normal- 
distributed. 


The Bernoulli distribution is perfect if we want 
, ,, . ^ . to flip a coin or get a yes/no answer. It emits 

bernoulli distribution r ° J 

only the values o or 1 and its only constructor 
parameter is the probability for the value of 1 . 


The discrete distribution is interesting if we 
only want a very limited, discrete set of values 
for which we want to define the probability for 
every individual value. Its constructor takes a 

.. ^ ^ ^ . list of weights and will emit random numbers 

discrete_distribution ° 

with probabilities depending on their weight. If 
we want to model randomly distributed blood 
groups, of which there are only four different 
ones that have specific probabilities, then this 
engine is a perfect match. 



Chapter 9. Parallelism and Concurrency 


In this chapter, we will cover the following recipes: 

• Automatically parallelizing code that uses standard algorithms 

• Putting a program to sleep for specific amounts of time 

• Starting and stopping threads 

/V 

• Performing exception-safe shared locking with std: :unique_iockA and 

std::shared_lock 

• Avoiding deadlocks with std:: scoped iock 

• Synchronizing concurrent std::cout use 

• Safely postponing initialization with std :: caii once 

• Pushing the execution of tasks into the background using std:: async 

• Implementing the producer/consumer idiom with 

std::condition_variable 

• Implementing the multiple producers/consumers idiom with 

std::condition_variable 

• Parallelizing the ASCII Mandelbrot Tenderer using std:: async 

• Implementing a tiny automatic parallelization library with std:: future 



Introduction 


Before C++11, C++ didn't have much support for parallelization. This does 
not mean that starting, controlling, stopping, and synchronizing threads was 
not possible, but it was necessary to use operating system-specific 

/V 

librariesA because threads are inherently operating system-related. 

With C++11, we got std:: thread, which enables basic portable thread 
control across all operating systems. For synchronizing threads, C++11 also 
introduced mutex classes and comfortable RAII-style lock wrappers. In 
addition to that, std:: condition_variable allows for flexible event 
notification between threads. 

Some other really interesting additions are std: :async and std:: future— we 
can now wrap arbitrary normal functions into std:: async calls in order to 
execute them asynchronously in the background. Such wrapped functions 
return std:: future objects that promise to contain the result of the function 
later, so we can do something else before we wait for its arrival. 

/V _ 

Another actually enormous improvementA to the STL are execution policies, 
which can be added to 69 of the already existing algorithms. This addition 
means that we can just add a single execution policy argument to the existing 
standard algorithm calls in our old programs and get parallelization without 
complex rewrites. 

In this chapter, we will go through all these additions in order to learn the 
most important things about them. Afterward, we'll have enough oversight 
ofA the parallelization support in the C++17 STL. We do not cover all the 
details, but the most important ones. The overview gained from this book 
helps inA quickly understanding the rest of the parallel 
programmingA mechanisms, which youA can always look up in the C++ 17 
STL documentation online. 

Finally, this chapter contains two bonus recipes. In one recipe, we will 
parallelize the Mandelbrot ASCII Tenderer from Chapter 6 . Advance Use of 





STL Algorithms, with only minimal changes. In the last recipe, we will 

/V 

implement a tiny library thatA helps parallelizing complex tasks implicitly 
and automatically. 



Automatically parallelizing code that uses 
standard algorithms 


C++17 came with one really major extension for parallelism: execution 

/V 

policies for standard algorithms. Sixty nineA algorithms were extended to 
accept execution policies in order to run parallel on multiple cores, and even 
with enabled vectorization. 

For the user, this means that if we already use STL algorithms everywhere, 
we get a nice parallelization bonus for free. We can easily give our 
applications subsequent parallelization by simply adding a single execution 
policy argument to our existing STL algorithm calls. 

In this recipe, we will implement a simple program (with a not too serious 
use case scenario) that lines up multiple STL algorithm calls. While using 
these, we will see how easy it is to use C++17 execution policies in order to 
let them run multithreaded. In the last subsections of this section, we will 
have a closer look at the different execution policies. 

How to do it... 

/\ 

In this section, we willA write a program that uses some standard algorithms. 
The program itself is more of an example of how real-life scenarios can look 
than doing actual real-life work situation. While using these standard 
algorithms, we are embedding execution policies in order to speed the code 
up: 

1. First, we need to include some headers and declare that we use the 

/\ /V 

A stdA namespace. The execution header is a new one; it came with 
C++17: 

#include <iostream> 

#include <vector> 

#include <random> 





#include <algorithm> 
#include <execution> 

using namespace std; 


2. Just for the sake of the example, we'll declare a predicate function that 

/\ 

tells whetherA a number is odd. We will use it later: 

static bool odd(int n) { return n % 2; } 

3. Let's first define a large vector in our main function. We will fill it with 

/y 

a lot of dataA so that it takes some time to do calculations on it. The 
execution speed of this code will vary a lot , depending on the computer 
this code is executed on. Smaller/larger vector sizes might be better on 
different computers: 

int main () 

{ 

vector<int> d (50000000); 

/\ 

4. In order to get a lot of random data for the vector, let's instantiate A a 
random number generator along with a distribution and pack them up in 

a callable object. If this looks strange to you, please first have a look at 

/\ 

the recipes thatA deal with random number generators and distributions 

/V 

in Chap ter A 8 . Utility Classes : 

mt 19 937 gen; 

unitorm_int_distribution<int> dis(0, 100000); 

auto rand_num ([=] () mutable { return dis(gen); }); 

5. Now, let's use the std::generate algorithm to fill the vector with 
random data. There is a new C++17 version of this algorithm, which can 
take a new kind of argument: an execution policy. We put in std: :par 

here, which allows for automatic parallelization of this code. By doing 

/\ 

this, we allow for multiple threads to A start filling the vector together, 

/\ 

which reduces the execution timeA if the computer has more than one 
CPU, which is usually the case with modern computers: 


generate (execution::par, begin(d), end(d) , rand_num); 



_ /\ 

6. The std:: sortA method should also already be familiar. The C++17 

/V 

version does also support an additional argument defmingA the 
execution policy: 

sort (execution: : par, begin(d), end(d)); 

7. The same applies to std: reverse: 

reverse (execution: : par, begin(d), end(d)); 

8. Then we use std:: count if to count all the odd numbers in the vector. 
And we can even parallelize that by just adding an execution policy 
again! 


auto odds (count_if (execution: : par, begin(d), end(d), o 

9. This whole program did not do any real scientific work, as we were just 
going to have a look on how to parallelize standard algorithms, but let's 
print something in the end: 

cout << (100.0 * odds / d.sizeO) 

<< "% of the numbers are odd.\n"; 

} 

10. Compiling and running the program gives us the following output. At 
this point, it is interesting to see how the execution speed differs when 

/V 

using the algorithms without an execution policy compared withA all 
the other execution policies. Doing this is left as an exercise for the 

/\ /V 

reader. Try it;A the available execution policies areA seq, par, and 
par vec. We should get different execution times for each of them: 

$ ./auto_parallel 

50.4% of the numbers are odd. 


How it works... 


Especially since this recipe did not distract us with any complicated real-life 
problem solution, we were able to fully concentrate on the standard library 
function calls. It is pretty obvious that the their parallelized versions are 



hardly different from the classic sequential ones. They only differ by 
oneA additional argument, which is the execution policy. 

/V 

Let's have a look at the invocations and answer threeA central questions: 

generate (execution::par, begin(d), end(d), rand_num); 
sort( execution: : par, begin(d) , end(d)); 
reverse( execution: : par, begin(d), end(d)); 

auto odds (count_if (execution::par, begin(d), end(d), odd)); 


Which STL algorithms can we parallelize this way? 

/\ 

Sixty nineA of the existing STL algorithms were upgraded to support 

/\ /V 

parallelism in the C++17 standard, and there are sevenA new ones that A also 
support parallelism. While such an upgrade might be pretty invasive for the 
implementation, not much has changed in terms of their interface—they all 

/V 

got an additionalA ExecutionPoiicy&& policy argument, and that's it. This 
does not mean that we always have to provide an execution policy argument. 
It is just that they additionally support accepting an execution policy as their 
first argument. 

These are the 69 upgraded standard algorithms. There are also the seven new 

/V 

ones thatA support execution policies from the beginning (highlighted in 
bold): 


std::adjacent_differencestd::adjacent_findstd::all_ofstd::any_ofs 


Having these algorithms upgraded is great news! The more our old programs 
utilize STL algorithms, the easier we can add parallelism to them 
retroactively. Note that this does not mean that such changes make every 
program automatically N times faster because multiprogramming is quite a 
bit more complex than that. 

However, instead of designing our own complicated parallel algorithms using 
std:: thread, std:: async, or by including external libraries, we can now 
parallelize standard tasks in a very elegant, operating system-independent 



way. 


How do those execution policies work? 

The execution policy tells which strategy we allow for the automatic 
parallelization of our standard algorithm calls. 

The following three policy types exist in the std:: execution namespace: 

Policy Meaning 

The algorithm has to be executed in a 
sequential form similar to the original 

sequenced poi icy algorithm without an execution policy. The 

globally available instance has the name 

std:: execution::seq. 

The algorithm may be executed with 
multiple threads that share the work in a 
paraiiei poiicy parallel fashion.A The globally available 

instance has the name 

std::execution::par. 

The algorithm may be executed with 

/\ 

multiple threads sharingA theA work. In 

/V 

addition to that, it is permissibleA to 
vectorize the code. In this case, container 
paraiiei unsequenced poiicy access can be interleaved between threads 

/v 

andA also within the same thread due to 

/V 

vectorization.A The globally available 
instance has the name 

std::execution::par_unseq. 



The execution policies imply specific constraints for us. The stricter the 
specific constraints, the more parallelization strategy measures we can allow: 

• All element access functions used by the parallelized algorithm must not 
cause deadlocks or data races 

• In the case of parallelism and vectorization, all the access functions must 
not use any kind of blocking synchronization 

As long as we comply with these rules, we should be free from bugs 
introduced by using the parallel versions of the STL algorithms. 

Note 

Note that just using parallel STL algorithms correctly does not 
always lead to guaranteed speedup. Depending on the problem 
we try to solve, the problem size, and the efficiency of our data 
structures and other access methods, measurable speedup will 
vary very much or not occur at all. Multiprogramming is still 
hard. 


What does vectorization mean? 

/V 

Vectorization is a feature that A both the CPU and the compiler need to 
support. Let's have a quick glance at a simple example to briefly understand 
what vectorization is and how it works. Imagine we want to sum up numbers 
from a very large vector. A plain implementation of this task can look like 
this: 

std::vector<int> v {1, 2, 3, 4, 5, 6, 7 /*...*/}; 
int sum {std::accumulate(v.begin(), v.end(), 0)}; 

The compiler will eventually generate a loop from the accumulate call, 

/V 

whichA could look like this: 


int sum {0 } ; 



for (size_t i {0}; i < v.size(); ++i) { 

sum += v [i]; 

} 


Proceeding from this point, with vectorization allowed and enabled, the 
compiler could then produce the followingA code. The loop does four 
accumulation steps in one loop step and also does four times fewer iterations. 
For the sake of simplicity, the example does not deal with the remainder if 

/V 

the vector does not containA n * 4 elements: 

int sum {0} ; 

for (size_t i {0}; i < v.size() / 4; i += 4) { 

sum += v[i] + v[i+l] + v[i + 2] + v[i + 3] ; 

} 

// if v.size() / 4 has a remainder, 

// real code has to deal with that also. 

/v 

Why should it do this? Many CPUs provide instructions thatA can perform 

/V 

mathematical operations such as A sum += v[i] + v[i+i] + v[i + 2] + 

/\ /\ 

v [ i + 3 ] ; A in just one step. PressingA as many mathematical operations 
into as few instructions as possible is the target because this speeds up the 
program. 

/\ 

Automatic vectorization is hardA because the compiler needs to understand 
our program to some degree in order to make our program faster but without 

/V 

tampering with itsA correctness. At least, we can help the compiler by using 
standard algorithms as often as possible because those are easier to grasp for 
the compiler than complicated handcrafted loops with complex data flow 
dependencies. 



Putting a program to sleep for specific 
amounts of time 


A nice and simple possibility to control threads came with C++11. It 
introduced the this thread namespace, which includes functions that affect 

/V 

only the caller thread. It contains twoA different functions that allow putting 
a thread to sleep for a certain amount of time, so we do not need to use any 
external or operating system-dependent libraries for such tasks any longer. 

In this recipe, we concentrate on how to suspend threads for a certain amount 
of time, or how to put them to sleep. 

How to do it... 


We will write a short program that just puts the main thread to sleep for 
certain amounts of time: 

1. Let's first include all the needed headers and declare that we'll use the 

/\ /V 

std andA chrono iiteraisA namespaces. 

/v /V 

TheA chrono iiteraisA namespace contains handy abbreviations for 
creating time-span values: 

#include <iostream> 

#include <chrono> 

#include <thread> 

using namespace std; 

using namespace chrono_literals; 

2. Let's immediately put the main thread to sleep for 5 seconds and 300 
milliseconds. Thanks to chrono literals, we can express this in a very 
readable format: 


int main() 
{ 




cout << "Going to sleep for 5 seconds" 
" and 300 milli seconds.\n"; 


this_thread::sleep_for(5s + 300ms); 

3. The last sleep statement was relative. We can also express absolute 
sleep requests. Let's sleep until the point in time, which is now plus 3 
seconds: 


cout << "Going to sleep for another 3 seconds.\n"; 

this_thread::sleep_until( 

chrono::high_resolution_clock::now() + 3s); 

4. Before quitting the program, let's print something else to signal the end 
of the second sleep period: 

cout << "That's it.\n"; 

} 

5. Compiling and running the program yields the following results. Linux, 

/\ 

Mac, and otherA UNIX-like operating systems provide the time 
command, which accepts another command in order to execute it and 
stop the time it takes. Running our program with time shows that it ran 
8.32 seconds, which is roughly the 5 . 3 and 3 seconds we let our 
program sleep. When running the program, it is possible to count the 
time between the arrival of the printed lines on the terminal: 

$ time ./sleep 

Going to sleep for 5 seconds and 300 milli seconds. 

Going to sleep for another 3 seconds. 

That's it. 

real0m8.320s 
userOmO.005s 
sys 0m0.003s 


How it works... 


^ /V /V 

TheA sieep for and sieep untiiA functions have been added toA C++11 

/V /V 

and reside in theA std:: this threadA namespace. They block the current 



thread (not the whole process or program) for a specific amount of time. A 
thread does not consume CPU time while it is blocked. It is just put into an 
inactive state by the operating system. The operating system does, of course, 
remind itself of waking the thread up again. The best thing about thisA is that 

we do not need to care which operating system our program runs 

/\ _ 

onA because the STL abstracts this detail away from us. 

The this_thread: :sleep_forA function accepts a chrono: : duration value. 
In the simplest case, this is just is or 5s + 30 0ms, just like in our example 
code. In order to get such nice literals for time spans, we need to declare 

using namespace std::chrono_literals;. 

The this_thread: : sleep_untilA function accepts a chrono: : time_point 

instead of a time span. This is comfortable if we wish to put the thread to 
sleep until some specific wall clock time. 

The timing for waking up is only as accurate as the operating system allows. 
This will be generally accurate enough A with most operating systems, but it 
might become difficult if some application needs nanosecond-granularity. 

Another possibility to put a thread to sleep for a short time is 

/\ 

this thread:: yield. It accepts noA arguments, which means that we cannot 
know for how long the execution of a thread is placed back. The reason is 
that this function does not really implement the notion of sleeping or parking 
a thread. It just tells the operating system in a cooperative way that it can 
reschedule any other thread of any other process. If there are none, then the 
thread will be executed again immediately. For this reason, yield is often 
less useful than just sleeping for a minimal, but specified, amount of time. 



Starting and stopping threads 


Another addition that came with C++11 is the std:: thread class. It provides 
a clean and simple way to start and stop threads, without any need for 
external libraries or to know how the operating system implements this. It's 
all just included in the STL. 

In this recipe, we will implement a program that starts and stops threads. 
There are some minor details to knowA what to do with threads once they are 
started, so we will go through these too. 

How to do it... 

We will start multiple threads and see how our program behaves when we 
unleash multiple processor cores to execute parts of its code at the same time: 

1. At first, we need to include only two headers and then we declare that 

/V /V 

we use theA std and chrono iiteraisA namespaces: 

#include <iostream> 

#include <thread> 

using namespace std; 

using namespace chrono_literals; 

2. In order to start a thread, we need to be able to tell what code 

/V 

shouldA be executed by it. So, let's define a function that can be 
executed. Functions are natural potential entry points for threads. The 

/V 

example function accepts an argument,A i, which acts as the thread ID. 
This way we can tell which print line came from which thread later. 
Additionally, we use the thread ID to let all threads wait for different 
amounts of time, so we can be sure that they do not try to use cout at 

exactly the same time. If they did, that would garble the 

/\ 

output.A Another recipe in this chapter deals specifically with this 
problem: 




static void thread_with_param(int i) 

{ 

this_thread::sleep_for(lms * i); 

cout << "Hello from thread " << i << ' \n 1 ; 

this_thread::sleep_for(Is * i); 

cout << "Bye from thread " << i << '\n'; 

} 

/v 

3. In the main function, we can, just out ofA curiosity, print how many 
threads can be run at the same time, using 

std: : thread: :hardware concurrency. This depends On how many 
cores the machine really has and how many cores are supported by the 
STL implementation. This means that this might be a different number 
on every other computer: 

int main() 

{ 

cout << thread::hardware_concurrency() 

<< " concurrent threads are supported.\n"; 

4. Let's now finally start threads. With different IDs for each one, we start 
three threads. When instantiating a thread with an expression such 

/V 

asA thread t {f, x} , this leads to a call of f (x) by the new thread. 
This ,way we can give the thread with param functions different 

/v 

arguments A for each thread: 

thread tl {thread_with_param, 1}; 
thread t2 {thread_with_param, 2}; 
thread t3 {thread_with_param, 3}; 


5. Since these threads are freely running, we need to stop them again when 
they are done with their work. We do this using the join function. It will 

/V 

block the callingA thread until the thread we try to join returns: 

tl.j oin (); 
t2.j oin(); 

6. An alternative to joining is detaching. If we do not call j oin or detach, 

/\ /\ 

the whole applicationA will be terminatedA with a lot of smoke and 



noise as soon as the destructor of the thread object is executed. By 

/V 

calling detach, we tell thread that we really want to let threadA number 
3 to continue running, even after its thread instance is destructed: 

t3.detach (); 

7. Before quitting the main function and the whole program, we print 
another message: 

cout << "Threads joined.\n"; 

} 

8. Compiling and running the code shows the following output. We can see 
that my machine has eight CPU cores. Then, we see the hello messages 
from all the threads, but the bye messages only from the two threads we 
actually joined. Thread 3 is still in its waiting period of 3 seconds, but 
the whole program does already terminate after the second thread has 
finished waiting for 2 seconds. This way, we cannot see the bye message 
from thread 3 because it was simply killed without any chance for 
completion (and without noise): 

$ ./threads 

8 concurrent threads are supported. 

Hello from thread 1 
Hello from thread 2 
Hello from thread 3 
Bye from thread 1 
Bye from thread 2 
Threads joined. 

How it works... 


Starting and stopping threads is a very simple thing to do. Multiprogramming 
starts to be complicated where threads need to work together (sharing 
resources, waiting for each other, and so on). 

/V 

In order to start a thread, we first need some function that willA be executed 
by it. The function does not need to be special, as a thread could execute 
practically every function. Let's pin down a minimal example program that 
starts a thread and waits for its completion: 



void f(int i) { cout << i << '\n 1 ; } 


int main() 

{ 

thread t {f, 123 } ; 
t. join () ; 

} 

The constructor call of std:: thread accepts a function pointer or a callable 
object, followed by arguments that should be used with the function call. It is, 
of course, also possible to start a thread on a function that doesn't accept any 
parameters. 

/\ 

If the system has multiple CPU cores, then the threads can run parallelA and 
concurrently. What is the difference between parallel and concurrent? If the 
computer has only one CPU core, then there can be a lot of threads that run in 
parallel but never concurrently because one CPU core can only run one 
thread at a time. The threads are then run in an interleaved way where every 
thread is executed for some parts of a second, then paused, and then the next 
thread gets a time slice (for human users, this looks like they run at the same 
time). If they do not need to share a CPU core, then they can run 
concurrently, as in really at the same time. 

At this point, we have absolutely no control over the following details: 
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• TheA order in which the threads are interleaved when sharing a CPU 
core. 

• The priority of a thread, or which one is more important than the other. 

• The fact that threads are really distributed among all the CPU cores or if 
the operating system just pins them to the same core. It is indeed 
possible that all our threads run on only a single core, although the 
machine has more than 100 cores. 

Most operating systems provide possibilities to control also these facets of 
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multiprogramming, but such features are, at this point,A not included in the 
STL. 

However, we can start and stop threads and tell them when to work on what 
and when to pause. That should be enough for a large class of applications. 



/\ 

What we did in this section was we started threeA additional threads. 
Afterward, we joined most of them and detached the last one. Let's 

/V 

summarize in a simple diagram A what happened: 



Reading the diagram from top to the bottom, it shows one point in time where 
we split the program workflow to four threads in total. We started 

/V 

threeA additional threads that did something (namely waiting and printing), 
but after starting the threads, the main thread executing the main function 

/V 

remained without work. A 

Whenever a thread has finished executing the function it was started with, it 
will return from this function. The standard library then does some tidy up 
work that results in the thread being removed from the operating system's 
schedule, and maybe in its destruction, but we do not need to worry about it. 

The only thing we need to worry about is joining. When a thread calls 
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function A x. j oin () on another thread object, it is put to sleep until 

/\ 

threadA x returns. Note that we are out of luck if the thread is trapped in an 
endless loop! If we want a thread to continue living until it decides to 
terminate itself, we can call x . detach (). After doing so, we have no external 
control over the thread any longer. No matter what we decide—we must 





always join or detach threads. If we don't do one of the two, the destructor of 
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the thread objectA will call std:: terminate (), which leads to an abrupt 
application shutdown. 

The moment when our main function returns, the whole application is, of 

/V /V 

course, terminated. However,A at the same time, our detached thread,A t3, 
was still sleeping before printing its bye message to the terminal. The 
operating system didn't care—it just terminated our whole program without 
waiting for that thread to finish. This is something we need to consider. If 
that additional thread had to complete something important, we would have 
to make the main function wait for it. 



Performing exception safe shared locking 
with std::unique_lock and std::shared_lock 


Since the operation of threads is a heavily operating system support-related 
thing and the STL provides good operating system-agnostic interfaces for 
that, it is also wise to provide STL support for synchronization between 
threads. This way, we can not only start and stop threads without external 

/V 

librariesA but also synchronize them with abstractions from a single unified 
library: the STL. 

In this recipe, we will have a look at STL mutex classes and RAII lock 
abstractions. While we play around with some of them in our concrete recipe 
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implementation, we will also get an overview of moreA synchronization 
helpers that the STL provides. 

How to do it... 


We are going to write a program that uses an std:: shared mutex instance in 
its exclusive and shared modes and to A see what that means. Additionally, 
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we do not call the lock and unlock functions ourselvesA but do the locking 
with automatic unlocking using RAII helpers: 


1. First, we need to include all necessary headers. Because we use STL 
functions and data structures all the time together with time literals, we 
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declare that we use theA std and chrono iiteraiA namespaces: 


#include 

#include 

#include 

#include 


<iostream> 

<shared_mutex> 

<thread> 

<vector> 


using namespace std; 

using namespace chrono_literals; 


2. The whole program revolves around one shared mutex, so let's define a 




global instance for the sake of simplicity: 

shared_mutex shared_mut; 

3. We are going to use the std: : shared_lock and std : : unique_lock 

RAII helpers. In order to make their names appear less clumsy, we 
define short type aliases for them: 

using shrd_lck = shared_lock<shared_mutex>; 
using uniq_lck = unique_lock<shared_mutex>; 

4. Before beginning with the main function, we define two helper functions 
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that both try to lock the mutex inA exclusive mode. This function here 

/\ 

will instantiate aA unique loc k instance on the shared mutex. The 
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second constructor argumentA defer iock tells the object to keep the 
lock unlocked. Otherwise, its constructor would try to lock the mutex 
and then block until it succeeds. Then we call tryiock on the 
exclusive lock object. This call will return immediately and its 
boolean return value tells us if it got the lock or if the mutex was locked 
already somewhere else: 

static void print_exclusive() 

{ 

uniq_lck 1 {shared_mut, defer_lock}; 

if (1.try_lock()) { 

cout << "Got exclusive lock.\n"; 

} else { 

cout << "Unable to lock exclusively.\n"; 

} 

} 

5. The other helper function tries to lock the mutex in exclusive mode, too. 
It blocks until it gets the lock. Then we simulate some error case by 
throwing an exception (which carries just a plain integer number instead 
of a more complex exception object). Although this leads to an 
immediate exit of the context in which we hold a locked mutex, the 
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mutex will cleanly be released again. That isA because the destructor of 
unique lock will release the lock in any case by design: 


static void exclusive throw() 



{ 

uniq_lck 1 {shared_mut}; 

throw 123; 

} 

6. Now to the main function. First, we open up another scopeA and 
instantiate a shared iock instance. Its constructor immediately locks the 
mutex in shared mode. We will see what this means in the next steps: 

int main() 

{ 

{ 

shrd_lck sll {shared_mut}; 

cout << "shared lock once.\n"; 

/\ 

7. Now we open yet another scopeA and instantiate a second shared iock 
instance on the same mutex. We have twoA shared iock instances 
now, and they both hold a shared lock on the mutex. In fact, we could 
instantiate arbitrarily many shared iock instances on the same mutex. 

~ /V 

Then we call print_exclusive, which tries to lockA the mutex in 
exclusive mode. This will not succeed because it is locked in shared 
mode already: 

{ 

shrd_lck sl2 {shared_mut}; 
cout << "shared lock twice.\n"; 
print_exclusive(); 

} 

8. After leaving the latest scope, the destructor of the shared_iocksi2 
releases its shared lock on the mutex. The print_exclusive function 
will again fail because the mutex is still in shared lock mode: 

cout << "shared lock once again.\n"; 

print_exclusive(); 


} 

cout << "lock is free.\n"; 



9. After leaving also the other scope, all shared iock objects are 

destroyed, and the mutex is in unlocked state again. Now we can finally 

lock the mutex in exclusive mode. Let's do this by calling 
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exciusive_throwA and then print_exciusive. Remember that we 
throw an exception in exciusive throw. But because unique iock is an 
RAII object that gives us exception safety, the mutex will be unlocked 
again no matter how we return from exclusive_throw. This way 
print_exciusive will not block on an erroneously still locked mutex: 

try { 

exclusive_throw() ; 

} catch (int e) { 

cout << "Got exception " << e << 1 \n 1 ; 

} 

print_exclusive(); 

} 

10. Compiling and running the code yields the following output. The first 
two lines show that we got the two shared lock instances. Then the 
print_exciusive function fails to lock the mutex in exclusive mode. 
After leaving the inner scope and unlocking the second shared lock, the 
print_exciusive function still fails. After leaving the other scope too, 
which finally released the mutex again, exciusive throw and 
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print_exciusiveA are finally able to lock the mutex: 

$ ./shared_lock 
shared lock once, 
shared lock twice. 

Unable to lock exclusively, 
shared lock once again. 

Unable to lock exclusively, 
lock is free. 

Got exception 123 
Got exclusive lock. 


How it works... 


When looking at the C++ documentation, it is at first a little confusing that 



there are different mutex classes and RAII lock-helpers. Before looking at our 
concrete code sample, let us summarize what the STL has available for us. 

Mutex classes 

The term mutex stands for mutual exclusion. In order to prevent that 
concurrently running threads alter the same object in a non-orchestrated way 
that might lead to data corruption, we can use mutex objects. The STL 
provides different mutex classes with different specialties. They all have in 
common that they have a lock and an unlock method. 

Whenever a thread is the first one to call lock () on a mutex that was not 
locked before, it owns the mutex. At this point, other threads will block on 
their lock calls, until the first thread calls unlock again, std::mutex can do 
exactly this. 

There are many different mutex classes in the STL: 


Type name 


Description 


Standard mutex with a lock and an unlock method, 
mutex Provides an additional nonblocking try iock 

method. 


Same as mutex, but provides additional 

timed_mutex try_lock_for and try_lock_until methods that 

allow for timing out instead of blocking forever. 

Same as mutex, but if a thread locked an instance of 
it already, it can call lock multiple times on the 
recursive mutex same mutex object without blocking. It is released 

after the owning thread called unlock as often as it 
called lock. 



Provides the features of both timed_mutex and 
recursive timed mutex recursive mutex. 


This mutex is special in that regard, that it can be 
locked in exclusive mode and in shared mode. In 
exclusive mode, it shows the same behavior as the 
standard mutex class. If a thread locks it in shared 
mode, it is possible for other threads to lock it in 
shared mutex shared mode, too. It will then be unlocked as soon 

as the last shared mode lock owner releases it. 
While a lock is locked in shared mode, it is not 
possible to obtain exclusive ownership. This is very 
similar to the behavior of shared ptr, only that it 
does not manage memory, but lock ownership. 


, . ^ ^ Combines the features of shared mutex and 
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timed mutex for both exclusive and shared mode. 


Lock classesA 

/\ 

Everything is nice and easy as long as threads do justA lock a mutex, access 
some concurrence protected object and unlock the mutex again. As soon as a 
forgetful programmer misses to unlock a mutex somewhere after locking it, 
or an exception is thrown while a mutex is still locked, things look ugly 
pretty quick. In the best case, the program just hangs immediately and the 
missing unlock call is identified quickly. Such bugs, however, are very 
similar to memory leaks, which also occur when there are missing 

/V 

explicit A delete calls. 

When regarding memory management, we have unique ptr, shared ptr 
and weak ptr. Those helpers provide very convenient ways to avoid memory 
leaks. Such helpers exist for mutexes, too. The simplest one is 
std:: lock guard. It can be used as follows: 



void critical_function() 

{ 

lock_guard<mutex> 1 {some_mutex}; 

// critical section 

} 

/\ 

lock guardA element's constructor accepts a mutex, on which it calls lock 
immediately. The whole constructor call will block until it obtains the lock on 
the mutex. Upon destruction, it unlocks the mutex again. This way it is hard 

/V 

to get the lock/unlock cycle wrongA because it happens automatically. 

The C++17 STL provides the following different RAII lock-helpers. They all 

accept a template argumentA that shall be of the same type as the mutex 

/\ 

(although, since C++17, the compiler can deduce that typeA itself): 


Name 


Description 


lock_guard 


This class provides nothing else than a constructor and a 
destructor, which lock and unlock a mutex. 


Similar to lock guard, but supports arbitrarily many mutexes 
scoped_iocki n its constructor. Will release them in opposite order in its 
destructor. 


Locks a mutex in exclusive mode. The constructor also accepts 
arguments that instruct A it to timeout instead of blocking 
, forever on the lock call. It is also possible to not lock the mutex 

unique lock A 

at all, or to assume that it is locked already, or to only try 
locking the mutex. Additional methods allow to lock and 
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unlock the mutex during the unique loc kA lock's lifetime. 


, . n Same as unique lock, but all operations are applied on the 
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mutex in shared mode. 



While lock guard and scoped iock have dead-simple interfaces that only 
consist of constructor and destructor, unique iock and shared iock are 
more complicated, but also more versatile. We will see in later recipes of this 
chapter, how else they can be used if not for plain simple lock regions. 

Let's get back to the recipe code now. Although we only ran the code in 
single thread context, we have seen how it is meant to use the lock helpers. 
The shrd_lck type alias stands for shared_lock<shared_mutex>A and 
allows us to lock an instance multiple times in shared mode. As long as sii 
and si 2 exist, no print_exclusive call is able to lock the mutex in exclusive 
mode. This is still simple. 

Now let's get to the exclusively locking functions that came later in the main 
function: 


int main() 
{ 

{ 


shrd_lck sll {shared_mut}; 

{ 

shrd_lck sl2 {shared_mut}; 

print exclusive(); 

} 

print_exclusive() ; 

} 


try { 

exclusive_throw() ; 

} catch (int e) { 

cout << "Got exception " << e << '\n 1 ; 

} 

print_exclusive() ; 

} 

One important detail is that after returning from exclusive_throw, the 
print exciusive function is able to lock the mutex again, although 
exclusive_throw did not exit cleanly due to the exception it throws. 

/\ 

Let's have another look at printexciusiveA because it used a strange 
constructor call: 



{ 


} 


uniq_lck 1 {shared_mut, defer_lock}; 

if (1.try_lock()) { 

/ / ... 

} 


/\ 

We did not only provide shared mutA but also defer iock as constructor 
arguments for unique iock in this procedure, defer iock is an empty global 
object that can be used to select a different constructor of unique loc k that 
simply does not lock the mutex. By doing so, we are able to call 
l. tryiock () later, which does not block. In case the mutex is locked 
already, we can do something else. If it was indeed possible to get the lock, 
we still have the destructor tidying up after us. 



Avoiding deadlocks with std::scoped_lock 


/V 

If deadlocks had occurred in road traffic, they would haveA looked like the 
following situation: 



In order to get the traffic flow going again, we either need a large crane that 
randomly picks one car from the center of the street intersection and removes 
it. If that is not possible, then we need enough drivers to be cooperative. The 
deadlock can be solved by all drivers in one direction driving several meters 
backwards, making space for the other drivers to continue. 

In multithreaded programs, such situations, of course, need to be avoided 
strictly by the programmer. It is however too easy to fail in that regard when 
the program is really complex. 

In this recipe,A we are going to write code which intentionally provokes a 
deadlock situation.A Then we will see how to write code that acquires the 
same resources that led the other code into a deadlock, but use the new STL 
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lock classA std:: scoped iock that came with C++17, in order to avoid this 

/V 

mistake. A 


How to do it... 


The code of this section contains two pairs of functions that ought to be 
executed by concurrent threads, and that acquire two resources in form of 
mutexes. One pair provokes a deadlock and the other avoids it. In the main 
function, we are going to try them out: 

1. Let's first include all needed headers and declare that we use namespace 

std and chrono_literals! 

#include <iostream> 

#include <thread> 

#include <mutex> 

using namespace std; 

using namespace chrono_literals; 

2. Then we instantiate two mutex objects which we need in order to run 
into a deadlock: 

mutex mut a; 
mutex mut b; 


3. In order to provoke a deadlock with two resources, we need two 

functions. One function tries to lock mutex A and then mutex B, while 
the other function will do that in the opposite order. By letting both 
functions sleep a bit between the locks, we can make sure that this code 
blocks forever on a deadlock. (This is for demonstration purposes. A 
program without some sleep lines might run successfully without a 

/V 

deadlock sometimes if we start it repeatedly.jA Note that we do not use 
the 1 \n 1 character in order to print a line break, but we use endi. endi 
does not only perform a line break but also flushes the stream buffer of 
cout, so we can be sure that prints are not bunched up and postponed: 

static void deadlock_func_l() 

{ 

cout << "bad fl acquiring mutex A..." << endi; 



lock_guard<mutex> la {mut_a}; 
this_thread::sleep_for(100ms); 

cout << "bad fl acquiring mutex B..." << endl; 
lock_guard<mutex> lb {mut_b}; 

cout << "bad fl got both mutexes." << endl; 

} 

4. As promised in the last step, deadiock_func_2 looks exactly same as 
deadiock func i, but it locks mutex A and B in the opposite order: 

static void deadlock_func_2() 

{ 

cout << "bad f2 acquiring mutex B..." << endl; 
lock_guard<mutex> lb {mut_b}; 

this_thread::sleep_for(100ms); 

cout << "bad f2 acquiring mutex A..." << endl; 
lock_guard<mutex> la {mut_a}; 

cout << "bad f2 got both mutexes." << endl; 

} 


5. Now we write a deadlock-free variant of those two functions we just 
implemented. They use the class scoped iock, which locks all mutexes 
we provide as constructor arguments. Its destructor unlocks them again. 
While locking the mutexes, it internally applies a deadlock avoidance 
strategy for us. Note that both functions still use mutex A and B in 
opposite order: 

static void sane_func_l() 

{ 

scoped_lock 1 {mut_a, mut_b}; 

cout << "sane fl got both mutexes." << endl; 

} 

static void sane_func_2() 

{ 

scoped_lock 1 {mut_b, mut_a}; 


cout << "sane f2 got both mutexes." << endl; 



} 


6. In the main function, we will go through two scenarios. First, we use the 
sane functions in multithreaded context: 


int main() 

{ 

{ 

thread tl {sane_func_l}; 
thread t2 {sane_func_2}; 

11.j oin (); 
t2.j oin(); 

} 

7. Then we use the deadlock-provoking functions that do not utilize any 
deadlock avoidance strategy: 

{ 

thread tl {deadlock_func_l}; 
thread t2 {deadlock_func_2}; 

11.j oin(); 
t2.j oin (); 

} 

} 

8. Compiling and running the program yields the following output. The 
first two lines show that the sane locking function scenario works and 
both functions return without blocking forever. The other two functions 
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run into a deadlock. We can tell that this is a deadlockA because we see 
the print lines that tell that the individual threads try to lock mutexes A 
and B and then wait forever. Both do not reach the point where they 
successfully locked both mutexes. We can let this program run for 
hours, days, and years, and nothing will happen. This application needs 
to be killed from outside, for example by pressing the keys Ctrl + C: 

$ ./avoid_deadlock 
sane fl got both mutexes 
sane f2 got both mutexes 
bad f2 acquiring mutex B... 
bad fl acquiring mutex A... 
bad fl acquiring mutex B... 



bad f2 acquiring mutex A... 


How it works... 

By implementing code that willfully causes a deadlock, we've seen how 
quick such an unwanted scenario can happen. In a large project, where 
multiple programmers write code that needs to share a common set of mutex- 
protected resources, all programmers need to comply with the same 
order A when locking and unlocking mutexes. While such strategies or rules 
are really easy to follow, they are also easy to forget. Another term for this 
problem is lock order inversion. 

scoped iock is a real help in such situations. It came with C++17 and works 
the same way as lock guard and unique iock work: its constructor 
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performsA the locking, and its destructor the unlocking of a mutex, 
scoped lock's specialty is that it can do this with multiple mutexes. 

scoped iock uses the std:: lock function, which applies a special algorithm 

that performs a series of tryiock calls on all the mutexes provided, in order 
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to preventA deadlocking. Therefore it is perfectly safe to use scoped iock or 
call std:: lock on the same set of locks, but in different orders. 



Synchronizing concurrent std::cout use 


One inconvenience in multithreaded programs is that we must practically 
secure every data structure they modify, with mutexes or other measures that 
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protect from uncontrolledA concurrent modification. 

One data structure that is typically used very often for printing is std:: cout. 
If multiple threads access cout concurrently, then the output will appear in 
interesting mixed patterns on the terminal. In order to prevent this, we would 
need to write our own function that prints in a concurrency-safe fashion. 

We are going to learn how to provide a cout wrapper that consists of minimal 

/V 

code itself and that is as comfortable to use as cout.A 


How to do it... 


In this section, we are going to implement a program that prints to the 
terminal concurrently from many threads. In order to prevent garbling of the 
messages due to concurrency, we implement a little helper class that 
synchronizes printing between threads: 


1. As always, the includes come first: 


#include 

#include 

#include 

#include 

#include 


<iostream> 

<thread> 

<mutex> 

<sstream> 

<vector> 


using namespace std; 


2. Then we implement our helper class, which we call pcout. The p stands 
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for parallel A because it works in a synchronized way for parallel 
contexts. The idea is that pcout publicly inherits from stringstream. 
This way we can use operator« on instances of it. As soon as a pcout 
instance is destroyed, its destructor locks a mutex and then prints the 




content of the stringstream buffer. We will see how to use it in the 
next step: 

struct pcout : public stringstream { 
static inline mutex cout_mutex; 

~pcout() { 

lock_guard<mutex> 1 {cout_mutex}; 
cout << rdbuf(); 
cout.flush (); 

} 

}; 


3. Now let's write two functions that can be executed by additional threads. 
Both accept a thread ID as arguments. Then, their only difference is that 
the first one simply uses cout for printing. The other one looks nearly 
identical, but instead of using cout directly, it instantiatesA pcout. This 
instance is a temporary object that lives only exactly for this line of 
code. After all operator« calls have been executed, the internal string 
stream is filled with what we want to print. ThenA pcoutA instance's 
destructor is called. We have seen what the destructor does: it locks a 
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specific mutex all pcout instances share alongA and prints: 

static void print_cout(int id) 

{ 

cout << "cout hello from " << id << 1 \n 1 ; 

} 

static void print_pcout(int id) 

{ 

pcout{} << "pcout hello from " << id << '\n 1 ; 

} 

4. Let's try it out. First, we are going to use print cout, which just uses 
cout for printing. We start 10 threads which concurrently print their 
strings and wait until they fini sh: 

int main() 

{ 

vector<thread> v; 

for (size_t i {0}; i < 10; ++i) { 

v.emplace back(print cout, i); 



} 


for (auto &t : v) { t.join(); } 

5. Then we do the same thing with the print pcout function: 

cout << "=====================\n"; 

v.clear(); 

for (size_t i {0}; i < 10; ++i) { 

v.emplace_back(print_pcout, i); 

} 

for (auto &t : v) { t.join(); } 

} 

6. Compiling and running the program yields the following result. As we 
see, the first 10 prints are completely garbled. This is how it can look 
like when cout is used concurrently without locking. The last 10 lines of 
the program are the print pcout lines which do not show any signs of 
garbling. We can see that they are printed from different 
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threadsA because their order appears randomized every time when we 
run the program again: 





$ ./sync_cout 
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How it works... 


Ok, we've built this "cout wrapper" that automatically serializes concurrent 
printing attempts. How does it work? 
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Let's do the same steps our pcout helper does in aA manual manner without 
any magic. First, it instantiates a string stream and accepts the input we feed 
into it: 

stringstream ss; 

ss << "This is some printed line " << 123 << '\n 1 ; 

Then it locks a globally available mutex: 

{ 

lock_guard<mutex> 1 {cout_mutex}; 

In this locked scope, it accesses the content of string stream ss, prints it, and 



releases the mutex again by leaving the scope. The cout. flush () line tells 

the stream object to print to the terminal immediately. Without this line, 
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aA program might run faster because multiple printed lines can be bunched 
up and printed in a single run later. In our recipes, we will like to see all 
output lines immediately, so we use the flush method: 

cout << ss.rdbuf(); 
cout.flush (); 

} 

/V /V 

Ok, this is simple enoughA but tedious to writeA if we have to to the same 

/V 

thing again and again. We can shorten down the stringstreamA instantiation 
as follows: 

stringstream{} << "This is some printed line " << 123 << 1 \n 1 ; 

This instantiates a string stream object, feeds everything we want to print into 
it and then destructs it again. The lifetime of the string stream is reduced to 
just this line. Afterward, we cannot print it any longer, because we cannot 
access it. Which code is the last that is able to access the stream's content? It 
is the destructor of stringstream. 

We cannot modify stringstream instance's member methods, but we can 
extend them by wrapping our own type around it via inheritance: 

struct pcout : public stringstream { 

~pcout() { 

lock_guard<mutex> 1 {cout_mutex}; 
cout << rdbuf(); 
cout.flush (); 

} 

}; 


This class is still a string stream and we can use it like any other string 
stream. The only difference is that it will lock a mutex and print its own 

/V 

buffer using cout.A 

/\ 

We also moved the cout mutex object into struct pcoutA as a static instance 
so we have both bundled in one place. 



Safely postponing initialization with 
std::call once 


Sometimes we have specific code sections that can be run in 

/V 

parallelA context by multiple threads with the obligation that some setup 
code must be executed exactly once before executing the actual functions. A 

/V 

simple solution is to just execute the existingA setup function before the 

/V 

program enters a state from which parallelA code can be executed from time 
to time. 

The drawbacks of such an approach are the following ones: 

• If the parallel function comes from a library, the user must not forget to 
call the setup function. That does not make the library easier to use. 

• If the setup function is expensive in some way, and it might not even 
need to be executed in case the parallel functions that need this setup are 
not even always used, then we need code that decides when/if to run it. 

In this recipe, we will have a look at std:: caii once, which is a helper 
function that solves this problem for us in a simple to use and 

/V 

elegantA implicit way. 

How to do it... 


We are going to write a program that starts multiple threads with exactly the 
same code. Although they are programmed to execute exactly the same code, 
our example setup function will only be called once: 

1. First, we need to include all the necessary headers: 

#include <iostream> 

#include <thread> 

#include <mutex> 

#include <vector> 



using namespace std; 


2. We are going to use std: : caii once later. In order to use it, we need an 
instance of once f lag somewhere. It is needed for the synchronization 
of all threads that use caii once on a specific function: 

once_flag callflag; 


3. The function which must be only executed once is the following one. It 
just prints a single exclamation mark: 

static void once_print() 

{ 

cout << '!'; 

} 

4. All threads will execute the print function. The first thing we do is 
Calling the function once_print through the function std: : caii once. 
caiionce needs the variable call flag we defined before. It will use it 
to orchestrate the threads: 

static void print(size_t x) 

{ 

std::call_once(callflag, once_print); 
cout << x; 

} 

/\ 

5. Ok, let's now start 10 threads which all use the print function:A 

int main() 

{ 

vector<thread> v; 

for (size_t i {0}; i < 10; ++i) { 

v.emplace_back(print, i) ; 

} 

for (auto &t : v) { t.join(); } 
cout << 1 \n 1 ; 

} 


6. Compiling and running yields the following output. First, we see the 



exclamation mark from the once print function. Then we see all thread 
IDs. caii_once did not only make sure that once print was only called 
once. Additionally, it synchronized all threads, so that no ID is printed 

before once_print was executed: 

$ ./call_once 
!1239406758 

How it works... 


std: caii once works like a barrier. It maintains access to a function (or a 
callable object). The first thread to reach it gets to execute the function. Until 
it has finished, any other thread that reaches the caii once line is blocked. 
After the first thread returns from the function, all other threads are released, 
too. 

/V 

In order to organize this littleA choreography, a variable is needed from 
which the other threads can determine if they must wait and when they are 
released again. This is what our variableA oncefiag calif lag; A is for. 

/V /V 

Every caii onceA line also needs a once fiagA instance as the argument 
prepending the function that shall be called only once. 

Another nice detail is: If it happens, that the thread which is selected to 
execute the function in caii_on cefails because some exception is thrown, 
then the next thread is allowed to execute the function again. This happens in 
the hope that it will not throw an exception the next time. 



Pushing the execution of tasks into the 
background using std::async 


Whenever we want some code to be executed in the background, we can 
simply start a new thread that executes this code. While this happens, we can 
do something else and then wait for the result. It's simple: 

std::thread t {my_function, argl, arg2, ...}; 

// do something else 

t.join(); // wait for thread to finish 

But then the inconvenience starts: t. j oin () does not give us the return value 
of my function. In order to get at that, we need to write a function that calls 
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my_f unction A and stores its return value in some variable that is also 
accessible for the first thread in which we started the new thread. If such 
situations occur repeatedly, then this represents quite a bunch of boilerplate 
code we have to write again and again. 

/V 

Since C++11, we have std :: a sync A which can do exactly this job for us and 
not only that. In this recipe, we are going to write a simple program that does 
multiple things at the same time using asynchronous function calls. 

/y 

AsA std:: async is a bit more powerful than that alone, we will have a closer 
look at its different facets. 


How to do it... 


We are going to implement a program that does multiple different things 
concurrently but instead of explicitly starting threads, we use std:: async and 
std::future: 


1. First, we include all necessary headers and declare that we use 

/\ /V 

theA stdA namespace: 


#include <iostream> 




#include <iomanip> 
#include <map> 
#include <string> 
#include <algorithm> 
#include <iterator> 
#include <future> 

using namespace std; 


2. We implement three functions which have nothing to do with 

A _ 

parallelismA but do interesting tasks. The first function accepts a string 
and creates a histogram of all characters occurring within that string: 

static map<char, size_t> histogram(const string &s) 

{ 

map<char, size_t> m; 

for (char c : s) { m[c] += 1; } 

return m; 

} 

3. The second function does also accept a string and returns a sorted copy 
of it: 


static string sorted(string s) 

{ 

sort(begin(s), end(s)); 
return s; 

} 


4. The third one counts how many vowels exist within the string it accepts 


static bool is_vowel(char c) 

{ 


} 


char vowels[] {"aeiou"}; 
return end(vowels) != 

find(begin(vowels) , 


end(vowels), 


static size_t vowels(const string &s) 

{ 


return count_if(begin (s), end(s), is_vowel); 


} 



5. In the main function, we read the whole standard input into a string. In 
order to not segment the input into words, we deactivate ios : : skipws. 
This way we get one large string, no matter how much white space the 
input contains. We use pop back on the resulting string afterward 
because we got one string terminating ' \ o ' character too much this way: 


int main() 

{ 

cin.unsetf(ios::skipws); 

string input {istream_iterator<char>{cin}, {}}; 

input.pop_back(); 

6. Now let's get the return values from all the functions we implemented 

before. In order to speed the execution up for very long input, we launch 
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them asynchronously. The std:: asyncA function accepts a policy, a 
function, and arguments for that function. We call histogram, sorted, 

/v 

and vowels with launch:: asyncA as a policy (we will see later what 
that means). All functions get the same input string as arguments: 


auto hist (async(launch::async, 

histogram, input)); 

auto sorted_str (async(launch::async, 

sorted, input)); 

auto vowel_count (async(launch::async, 

vowels, input)); 

7. The async calls return immediatelyA because they do not actually 
execute our functions. Instead, they set up synchronization structures 
which will obtain the results of the function calls later. The results are 

/V /V 

nowA being calculated concurrently by additionalA threads. In the 
meantime, we are free to do whatever we want, as we can pick up those 

/V 

values later. The return values hist, sorted str and vowei countA are 
of the types the functions histogram, sorted, and vowels return, but 

/v 

they wereA wrapped in a future type by std:: async. Objects of this 
type express that they will contain their values at some point in time. By 
using . get () on all of them, we can make the main function block until 
the values arrive, and then use them for printing: 

for (const auto &[c, count] : hist. getO) { 
cout << c << ": " << count << '\n'; 

} 



cout << "Sorted string: " 

<< quoted(sorted_str.get()) << '\n' 

<< "Total vowels: " 

<< vowel_count.get() << '\n'; 

} 

8. Compiling and running the code looks like the following. We use a short 
example string that does not really make it worth being parallelized, but 
for the sake of this example, the code is nevertheless executed 
concurrently. Additionally, the overall structure of the program did not 
change much compared to a naive sequential version of it: 

$ echo "foo bar baz foobazinga" | ./async 
: 3 
a: 4 
b: 3 
f: 2 
g: 1 
i: 1 
n: 1 
o: 4 
r: 1 
z : 2 

Sorted string: " aaaabbbffginoooorzz" 

Total vowels: 9 


How it works... 


If we would not have used std: : async the serial unparallelized code could 
have looked as simple as that: 

auto hist (histogram(input)); 

auto sorted_str (sorted( input)); 
auto vowel_count (vowels( input)); 

for (const auto &[c, count] : hist) { 
cout << c << ": " << count << '\n'; 

} 

cout << "Sorted string: " << quoted(sorted_str) << '\n'; 
cout << "Total vowels: " << vowel count << '\n'; 




The only thing we did in order to parallelize the code was the following. We 

/V 

wrapped the three function calls into async (launch:: async, ...) A calls. 
This way these three functions are not executed by the main thread we are 
currently running in. Instead, async starts new threads and lets them execute 
the functions concurrently. This way we get to execute only the overhead of 

/V 

starting another threadA and can continue with the next line of code, while all 
the work happens in the background: 


auto hist (async(launch::async , histogram, input)); 

auto sorted_str (async(launch :: async , sorted, input)); 
auto vowel_count (async(launch :: async , vowels, input)); 

for (const auto &[c, count] : hist. get()) { 
cout << c << " << count << '\n 1 ; 

} 

cout << "Sorted string: " 

<< quoted(sorted_str.get()) << '\n' 

<< "Total vowels: " 

<< vowel_count.get() << '\n'; 

/\ 

While histogramA for example, returns us a map instance, async (..., 
histogram, ...) does return us a map that was wrapped in a future object 
before. This future object is kind of an empty placeholder until the thread 
that executes the histogram function returns. The resulting map is then 
placed into the future object so we can finally access it. The get function 
then gives us access to the encapsulated result. 

Let's have a look at another minimal example. Consider the following code 
snippet: 


auto x (f(1, 2, 3)); 
cout << x; 

/v 

Instead of writing A the preceding code, we can also do the following: 

auto x (async(launch::async, f, 1, 2, 3) ) ; 

cout << X. get () ; 


That's basically it. Executing tasks in the background might have never been 
easier in standard C++. There is still one thing left to resolve: What does 
launch:: async mean? launch:: async is a flag that defines the launch 



/v 

policy. There are two policy flags which allow for threeA constellations: 


Policy choice 


Meaning 


, , The function is guaranteed to be executed by another 

launch::async ° J 

thread. 


The function is executed by the same thread, but later 
, , . . (lazy evaluation). Execution then happens when get or 

_L3."U.nOiT * * Q.O£OITTTOQ. '' * / ' A 

wait is called on the future. If none of both happens, the 
function is not called at all. 


launch 

launch 


Having both flags set, the STL's async implementation is 

3 s yno | 

deferred ^ ree to c h° ose which policy shall be followed. This is the 
default choiceA if no policy is provided. 


Note 

By just calling async (f, 1 , 2 , 3 ) without a policy argument, 
we automatically select both policies. The implementation of 
async is then free to choose which policy to employ. This means 
that we cannot be sure that another thread is started at all, or if 
the execution is just deferred in the current thread. 

There’s more... 

There is indeed one last thing we should know about. Suppose, we write 

/y 

codeA as follows: 


async(launch::async, f); 
async(launch::async, g); 



/v 

This might have the motivation of executing functions A f and g (we do not 
care about their return values in this example) in concurrent threads and then 

doing different things at the same time. While running such code, we will 

/\ 

notice that the codeA blocks on this calls, which is most probably not what 
we want. 

So why does it block? Isn't a sync all about nonblocking asynchronous calls? 
Yes it is, but there is one special peculiarity: if a future was obtained from an 

/V /V 

async call with theA launch:: asyncA policy, then its destructor performs a 

/v 

blocking wait. A. 

This means that both the async calls from this short example are 

/V 

blockingA because the lifetime of the futures they return ends in the same 
line! We can fix this by capturing their return values in variables with a 
longer lifetime. 



Implementing the producer/consumer idiom 
with std:: condition variable 


In this recipe, we are going to implement a typical producer/consumer 
program with multiple threads. The general idea is that there is one thread 
that produces items and puts them into a queue. Then there is another thread 
that consumes such items. If there is nothing to produce, the producer thread 
sleeps. If there is no item in the queue to consume, the consumer sleeps. 

Since the queue that both threads have access to is also modified by both 
whenever an item is produced or consumed, it needs to be protected by a 
mutex. 

/V 

Another thing to consider is: What does the consumer do A if there is no item 
in the queue? Does it poll the queue every second until it sees new items? 
That is not necessaryA because we can let the consumer wait for wakeup 

/V 

events that are triggeredA by the producer, whenever there are new items. 

C++11 provides a nice data structure called std:: condition variabie for 
this kind of events. In this recipe, we are going to implement a simple 
producer/consumer app that takes advantage of this. 

How to do it... 


We are going to implement a simple producer/consumer program which runs 
a single producer of values in its own thread, as well as a single consumer 
thread in another thread: 

1. First, we need to perform all the needed includes: 

#include <iostream> 

#include <queue> 

#include <tuple> 

#include Ccondition variable> 





#include <thread> 


using namespace std; 

using namespace chrono_literals; 

2. We instantiate a queue of simple numeric values and call it q. The 
producer will push values into it, and the consumer will take values out 
of it. In order to synchronize both, we need a mutex. In addition to that, 
we instantiate a condition variabiecv. The variable finished will be 
the producer's way to tell the consumer that no more values will follow: 

queue<size_t> q; 

mutex mut; 

condition_variable cv; 
bool finished {false}; 

3. Let's first implement the producer function. It accepts an argument 
items which limits the maximum number of items for production. In a 
simple loop, it will sleep 100 milliseconds for every item, which 
simulates some computational complexity. Then we lock the mutex that 
synchronizes access to the queue. After successful production and 
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insertion to the queue, we callA cv. notify aii () . This function wakes 
the consumer up. We will see later at the consumer side how this works: 

static void producer(size_t items) { 

for (size_t i {0}; i < items; ++i) { 

this_thread::sleep_for(100ms); 

{ 

lock_guard<mutex> lk {mut}; 
q.push (i) ; 

} 

cv.notify_all(); 

} 

4. After having produced all items, we lock the mutex again because we 
are going to change to set the finished bit. Then we call 

cv.notify_aii () again: 

{ 

lock_guard<mutex> lk {mut}; 
finished = true; 

} 

cv.notify all (); 



} 

5. Now we can implement the consumer function. It takes no 
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argumentsA because it will blindly consume until the queue runs empty. 
In a loop that is executed as long as finished is not set, it will first lock 
the mutex that protects both the queue and the finished flag. As soon as 
it has the lock, it calls cv.wait with the lock and a lambda expression as 
arguments. The lambda expression is a predicate that tells if the 
producer thread is still alive and if there is anything to consume in the 
queue: 

static void consumer() { 

while (!finished) { 

unique_lock<mutex> 1 {mut}; 

cv.wait(1, [] { return !q.empty() || finished; }); 

_ /\ 

6. The cv.wait callA unlocks the lock and waits until the condition 
described by the predicate function holds. Then, it locks the mutex again 
and consumes everything from the queue until it appears empty. If the 
producer is still alive, it will iterate through the loop again. Otherwise, it 
will terminate because finished is set, which is the producer's way to 
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signal that there are no furtherA items being produced: 

while (Iq.emptyO) { 

cout << "Got " << q.front() 

<< " from queue.\n"; 
q•pop (); 

} 

} 

} 

7. In the main function, we start a producer thread which produces 10 
items, and a consumer thread. Then we wait until their completion and 
terminate the program: 

int main() { 

thread tl {producer, 10}; 
thread t2 {consumer}; 
tl.j oin (); 
t2.j oin(); 

cout << "finished!\n"; 



} 


8. Compiling and running the program yields the following output. When 
the program is executed, we can see that there is some time (100 
milliseconds) between each line, because the production of items takes 
some time: 

$ ./producer_consumer 
Got 0 from queue. 

Got 1 from queue. 

Got 2 from queue. 

Got 3 from queue. 

Got 4 from queue. 

Got 5 from queue. 

Got 6 from queue. 

Got 7 from queue. 

Got 8 from queue. 

Got 9 from queue, 
finished! 

How it works... 

/V 

In this recipe, we simply started two threads. The first threadA produces 
items and puts them into a queue. The other takes items out of the queue. 
Whenever one of those threads touches the queue in any way, it locks the 
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common mutexA mut A which is accessible for both. This way we made sure 
that it cannot happen that both threads manipulate the queue's state at the 
same time. 

Apart from the queue and the mutex, we declared generally four variables 
that were involved in the producer-consumer thing: 

queue<size_t> q; 

mutex mut; 

condition_variable cv; 
bool finished {false}; 

The variable finished is easy to explain. It was set to true when the 
producer finished producing its fixed amount of items. When the consumer 
sees that this variable is true, it consumes the last items in the queue and 
stops consuming. But what is the condition variabiecv for? We used cv in 



two different contexts. One of the contexts was waiting for a specific 
condition , and the other was signaling that condition. 

The consumer side that waits for a specific condition looks like this. The 
consumer thread loops over a block that first locks mutex mut in a 

unique_lock. Then it calls cv.wait! 

while (!finished) { 

unique_lock<mutex> 1 {mut}; 

cv.wait(l, [] { return !q.empty() || finished; }); 

while (!q.empty()) { 

// consume 

} 

} 

This code is somewhat equivalent to the following alternative code. We will 
elaborate soon why it is not really the same: 

while (!finished) { 

unique_lock<mutex> 1 {mut}; 

while (q.empty() && !finished) { 

1.unlock(); 

1.lock(); 

} 

while (Iq.emptyO) { 

// consume 

} 

} 

This means that we generally first acquire the lock and then check what 
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scenario A we have: 

1. Are there items to consume? Then keep the lock, consume, release the 
lock, and start over. 

/V 

2. Else, if there are no consumable items A but the producer is still 

/V 

alive, A release the mutex to give the producer a chance of adding items 
to the queue. Then, try to lock it again in hope that the situation changes 
and we get to see situation 1. 



The real reason why the cv.wait line is not equivalent to the while 

/V 

(q. empty () & & ... ) construct is, that we cannot simplyA loop over a 
1 . unlock (); l. lock (); cycle. If the producer thread is inactive for some 
time, then this would lead to continuous locking and unlocking of the mutex, 
which makes no sense because it needlessly burns CPU cycles. 

An expression like cv.wait (lock, predicate) will wait until predicate () 

returns true. But it does not do this by continuously unlocking and locking 

lock. In order to wake a thread up that blocks on the wait call of a 

/\ 

condi tion_vari able A object, another thread has to call the not if y_one () or 
not if y_aii () method on the same object. Only then the waiting thread(s) 
is/are kicked out of their sleep in order to check if predicate () holds. 

The nice thing about the wait call checking the predicate is that if there is a 
spurious wakeup call, the thread will go to sleep immediately again. This 
means that it does not really harm the program flow (but maybe the 
performance) if we have too many notify calls. 

On the producer side, we just called cv.notifyaii o after the producer 
inserted an item to the queue and after it produced its last item and set the 
finished flag to true. This was enough to direct the consumer. 



Implementing the multiple 
producers/consumers idiom with 
std::condition variable 


Let's pick up the producer/consumer problem from the last recipe and make it 
a bit more complicated: We make multiple producers produce items and 
multiple consumers consume them. In addition to that, we define that the 
queue shall not exceed a maximum size. 

, /v 

ThisA way not only the consumers have to sleep from time to time if there 
are no items in the queue, but also the producers have to sleep from time to 
time when there are enough items in the queue. 

We are going to see how to solve this problem with multiple 

std:: condition_variable objects and will also use them in slightly 

different ways than in the last recipe. 

How to do it... 


In this section, we are going to implement a program just like in the recipe 
before, but this time with multiple producers and multiple consumers: 

1. First, we need to include all needed headers and we declare that we use 
namespace std and chrono iiterais: 

#include <iostream> 

#include <iomanip> 

#include <sstream> 

#include <vector> 

#include <queue> 

#include <thread> 

#include <mutex> 

#include <condition_variable> 

#include <chrono> 




using namespace std; 

using namespace chrono_literals; 


2. Then we implement the synchronized printing helper from the other 
recipe in this chapter because we are going to do a lot of concurrent 
printing: 


struct pcout : public stringstream { 
static inline mutex cout mutex; 

~pcout() { 

lock_guard<mutex> 1 {cout_mutex}; 
cout << rdbuf(); 

} 

}; 


3. All producers write values into the same queue and all consumers will 
also take values out of this queue. In addition to that queue, we need a 
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mutex that protects both the queue and a flag that can tellA if the 
production was stopped at some point: 

queue<size_t> q; 
mutex q mutex; 

bool production_stopped {false}; 

4. We are going to employ two different condition variabies in this 
program. In the single producer/consumer recipe, we had a 

condition_variable telling that there are new items in the queue. In 
this case, we make it a bit more complicated. We want the producers to 
produce until the queue contains a certain stock amount of items. If that 
stock amount is reached, they shall sleep. This way the go consume 
variable can be used to wake up consumers which then, in turn, can 
wake up the producers with the go produce variable again: 


condition_variable go_produce; 
condition_variable go_consume; 

5. The producer function accepts a producer ID number, a total number of 
items to produce and a stock limit as arguments. It then enters its own 
production loop. There, it first locks the queue's A mutex A and unlocks it 
again in the go_produce. wait call. It waits for the condition that the 
queue size is below the stock threshold: 



static void producer(size_t id, size_t items, size_t stock) 

{ 

for (size_t i = 0; i < items; ++i) { 

unique_lock<mutex> lock(q_mutex); 
go_produce.wait(lock, 

[&] { return q.size() < stock; }); 

/v /\ 

6. After A the producer was A woken up, it produces an item and pushes it 
into the queue. The queue value is calculated from the expression id * 

100 + i. This way we can later see which producer produced 
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it A because the hundreds in the number are the producer ID. We also 
print the production event to the terminal. The format of the printing 
may look strange, but it will align nicely with the consumer output in the 
terminal later: 

q.push(id * 100 + i) ; 

pcout{} << " Producer " << id << " --> item " 

<< setw(3) << q.back() << ' \n' ; 

7. After production, we can wake up sleeping consumers. A sleeping 
period of 90 milliseconds simulates that producing items takes some 
time: 


go_consume.notify_all(); 
this_thread::sleep_for(90ms); 

} 

pcout{} << "EXIT: Producer " << id << '\n'; 

} 

8. Now to the consumer function that only accepts a consumer ID as an 
argument. It shall continue waiting for items if the production has not 
stopped, or the queue is not empty. If the queue is empty, but the 
production has not stopped, then it is possible that there might be new 
items soon: 


static void consumer(size_t id) 

{ 

while (!production_stopped || !q.empty()) { 

unique lock<mutex> lock(q mutex); 



9. After locking the queue mutex, we unlock it again in order to wait 
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onA the go consume event variable. The lambda expression argument 

describes that we want to return from the wait call when the queue 
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contains items. The second argumentA is tells that we do not want to 
wait forever. If it takes longer than 1 second, we want to drop out of the 
wait function. We can distinguish if the wait_for function returned 
because the predicate condition holds, or if we dropped out of it because 
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of a timeoutA because it will return false in case of the timeout. If there 
are new items in the queue, we consume them and print this event to the 
terminal: 


if (go_consume.wait_for(lock. Is, 

[] { return !q.empty(); })) { 

pcout{} << " item " 

<< setw(3) << q. front () 

<< " --> Consumer " 

<< id << '\n 1 ; 

q•pop(); 

10. After item consumption, we notify the producers and sleep for 130 
milliseconds to simulate that consuming items is also time-consuming: 


go_produce.notify_all(); 
this_thread::sleep_for(130ms); 

} 

} 

pcout{} << "EXIT: Producer " << id << 1 \n 1 ; 

} 

11. In the main function, we instantiate a vector for worker threads and 
another for consumer threads: 


int main() 

{ 

vector<thread> workers; 
vector<thread> consumers; 

12. Then we spawn three producer threads and five consumer threads: 

for (size_t i = 0; i < 3; ++i) { 

workers.emplace_back(producer, i, 15, 5); 


} 



for (size_t i = 0; i < 5; ++i) { 

consumers.emplace_back(consumer, i); 

} 

13. We first let the producer threads finish. As soon as all of them have 
returned, we set the production_stopped flag, which will lead the 
consumers to finish, too. We need to collect those and then we can quit 
the program: 


for (auto &t : workers) { t.join(); } 

productions topped = true; 

for (auto &t : consumers) { t.join(); } 

} 

14. Compiling and running the program leads to the following output. The 
output is very long, which is why it is truncated here. We can see that 
the producers go to sleep from time to time, and let the consumers eat up 
some items until they finally produce again. It is interesting to alter the 
wait times for producers/consumers, as well as manipulating the number 
of producers/consumers and stock items because this completely 
changes the output patterns: 

$ ./multi_pr°ducer_consumer 
Producer 0 --> item 0 
Producer 1 --> item 100 
item 0 --> Consumer 0 
Producer 2 --> item 200 
item 100 --> Consumer 1 
item 200 --> Consumer 2 
Producer 0 --> item 1 
Producer 1 --> item 101 
item 1 --> Consumer 0 

Producer 0 --> iteml4 
EXIT: Producer 0 
Producer 1 --> item 114 
EXIT: Producer 1 
iteml4 --> Consumer 0 
Producer 2 --> item 214 
EXIT: Producer 2 
item 114 --> Consumer 1 
item 214 --> Consumer 2 
EXIT: Consumer 2 



EXIT : 

Consumer 

3 

EXIT : 

Consumer 

4 

EXIT : 

Consumer 

0 

EXIT : 

Consumer 

1 


How it works... 


This recipe is an extension of the preceding recipe. Instead of synchronizing 
only one producer with one consumer, we implemented a program that 
synchronizes m producers with n consumers. On top of that, not only the 
consumers go to sleep if there are no items for them left, but also the 
producers go to sleep as soon as the item queue becomes too long. 

When multiple consumers wait for the same queue to fill up, then this would 

/V 

generally also work with the consumer code A from the one producer/one 
consumer scenario. As long as only one thread locks the mutex that protects 

/V 

the queue and then takes items out ofA it, the code is safe. It does 

/\ ^ 

not A matter how many threads are waiting for the lock at the same time. The 
same applies to the producers, as in both scenarios the only important thing is 
that the queue is never accessed by more than one thread at a time. 

So what makes this program really more complex than just running the one 
producer/one consumer example with more threads is the fact that we make 
the producer threads stop as soon as the item queue length reached a certain 
threshold. In order to meet that requirement, we implemented twoA different 
signals with their own condition_variable: 

1. The go produce signals the event that the queue is not completely filled 
to the maximum and the producers may fill it up again. 

2. The go consume signals the event that the queue reachedA its maximum 
length and consumers are free to consume items again. 

This way producers fill items into the queue and signal the go consume event 
to the consuming threads, which wait on the following line: 

if (go_consume. wait_f or (lock. Is, [] { return Iq.emptyO; })) { 

// got the event without timeout 




The producers, on the other hand, wait on the following line until they are 
allowed to produce again: 

go_produce.wait(lock, [&] { return q.size() < stock; }); 

One interesting detail is that we do not let consumers wait forever. In the 

/V 

go consume. wait for call, we additionally added a timeout argumentA of 1 

/V 

second. This is the exit mechanism for consumers: if the queue is A empty for 

/\ 

longerA than a second, maybe there are no active producers any longer. 

For the sake of simplicity, the code tries to keep the queue length always at 
the maximum. A more sophisticated program could let the consumer threads 
push a wake-up notification, only if the queue has only half the size of its 
maximum length. This way producers would be woken up before the queue 

/V 

runs empty again, but not unnecessarilyA earlier when there are still enough 
items in the queue. 

One situation that condition variabie solves elegantly for us is the 
following: If a consumer fires the go produce notification, there might be a 
horde of producers racing to produce the next item. If only one item is 
missing, then there will only be one producer producing it. If all producers 
would always produce an item as soon as the go produce event is fired, we 
would often see the case that the queue is filled above its allowed maximum. 

Let's imagine the situation that we have (max - l) items in the queue and 
want one new item produced so that the queue is filled up again. No matter if 

/V 

a consumer thread calls goproduce. not if y_one () A (which would wake up 
only one waiting thread) or goproduce. not if y_aii () (which wakes up all 
waiting threads), we have the guarantee that only one producer thread will 
exit the go produce. wait call, because, for all other producer threads, 

/V /V 

theA q. size () < stockA wait condition doesn't hold any longer as soon as 
they get the mutex after being woken up. 



Parallelizing the ASCII Mandelbrot 
renderer using std::async 


Remember the ASCII Mandelbrot renderer from Chapter 6 . Advanced Use of 
STL algorithms ? In this recipe, we willA make it use threads in order to speed 
its calculation time a bit up. 

/\ 

First, we will modify the A line in the original program that limits the number 
of iterations for every selected coordinate. This will make the program slower 
and its results more accurate than we can actually display on the terminal, 
but then we have a nice example target for parallelization. 

/v 

Then, we will apply minorA modifications to the program and see how the 
whole program runs faster. After those modifications, the program runs with 
std:: async and std:: future. In order to fully understand this recipe, it is 
crucial to understand the original program. 

How to do it... 

In this section, we take the ASCII Mandelbrot fractal renderer that we 
implemented in Chapter 6 . Advanced Use of STL Algorithms. First, we are 
going to make the calculation take much more time by incrementing the 
calculation limit. Then we get some speedup by doing only four little changes 
to the program in order to parallelize it: 

1. In order to follow the steps, it is best to just copy the whole program 
from the other recipe. Then follow the instructions in the following steps 
in order to do all needed adjustments. All differences from the original 

/V _ 

program are highlightedA in bold. The first change is an additional 

/\ 

header,A <future>: 


#include <iostream> 
#include <algorithm> 
#include <iterator> 






#include <complex> 
#include <numeric> 
#include <vector> 

tinclude <future> 


using namespace std; 

/v 

2. The A scaler and scaiedcmpix functions don't need any change; 


using cmplx = complex<double>; 


static auto 
{ 


scaler(int min_from, int max_from, 

double min to, double max to) 


const int w_from {max_from - min_from}; 
const double w_to {max_to - min_to}; 

const int mid_from {(max_from - min_from) / 2 + min_fro 
const double mid to {(max to - min to) / 2.0 + min to}; 


return [=] (int from) { 

return double(from - mid_from) / w_from * w_to + mi 

} ; 


template <typename A, typename B> 

static auto scaled_cmplx(A scaler_x, B scaler_y) 

{ 

return [=](int x, int y) { 

return cmplx{scaler_x(x), scaler_y(y)}; 

}; 

} 


3. In the function mandeibrot iterations, we are just going to increment 
the number of iterations in order to make the program a bit more 
computation-heavy: 


static auto mandelbrot_iterations(cmplx c) 

{ 

cmplx z {}; 

size_t iterations {0} ; 
const size_t max_iterations { 100000 }; 
while (abs(z) < 2 && iterations < max_iterations) { 

++iterations; 
z = pow(z, 2) + c; 



return iterations; 

} 

4. Then we have a part of the main function that does not need any change 
again: 

int main() 

{ 

const size_t w {100} ; 
const size_t h {40}; 

auto scale (scaled_cmplx( 

scaler(0, w, -2.0, 1.0), 
scaler(0, h, -1.0, 1.0) 


auto i_to_xy ([=](int x) { 

return scale(x % w, x / w); 

}) ; 

5. In the to_iteration_count function, we do not call 

mandeibrot_iterations (x_to_xy (x) ) directly any longer, but make 
the call asynchronous using std: : async: 

auto to iteration_count ([=](int x) { 
return async(launch :: async, 

mandelbrot_iterations, i_to_xy(x)); 

}) ; 

6. Before the last change, the function to iteration count returned us 

/V 

the number of iterations a specific coordinate needsA for the Mandelbrot 
algorithm to converge. Now it returns a future variable that will contain 
the same value later because it is computed asynchronously. Because of 
this, we need a vector that holds all the future values, so let's just add 
one. The output iterator we provide transform as the third argument 
must be the begin iterator of the new output vector r: 

vector<int> v (w * h); 

vector<future<size_t» r (w * h); 

iota(begin(v), end(v), 0); 
transform(begin(v), end(v), begin (r) , 
to iteration count); 



7. The accumulate call which did all the printing for us doesn't get size t 
values as its second argument any longer, but future<size_t> values. 
We need to adapt it to this type (if we had used autos as its type from 
the beginning then this would not even be necessary), and then we need 
to call x. get () where we just accessed x before, in order to wait for the 
value to arrive: 


auto binfunc ([w, n{0}] (auto output_it, future<size_t> 
mutable { 

*++output_it = (x.get() > 50 ? : ' '); 

if (++n % w == 0) { ++output_it = '\n' ; } 

return output_it; 

}) ; 


} 


accumulate(begin(r), end(r), 

ostream iterator<char>{cout}, binfunc); 


8. Compiling and running gives us the same output as before. The only 
interesting difference is the execution speed. If we increase the number 
of iterations for the original version of the program, too, then the 
parallelized version should compute faster. On my computer with four 
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CPU cores with hyperthreading (which results in 8 virtual cores),A I get 
different results with GCC and clang. The best speedup is 5.3, and the 
worst is 3.8. The results will also vary across machines, of course. 


How it works... 

It is crucial to understand the whole program first because then it is clear that 

/V 

all the CPU-intense work happensA in one line of code in the main function: 


transform(begin(v), end(v), begin(r), to_iteration_count); 


The vector v contains all the indices that are mapped to complex coordinates, 
which are then in turn iterated over with the Mandelbrot algorithm. The result 
of each iteration is saved in vector r. 

In the original program, this is the single line which consumes all the 
processing time for calculating the fractal image. All code that precedes it is 
just set up work and all code that follows it is just for printing. This means 



that parallelizing this line is key to more performance. 


One possible approach to parallelizing this is to break up the whole linear 
range from begin (v) to end (v) into chunks of the same size and distribute 
them evenly across all cores. This way all cores would share the amount of 
work. If we used the parallel version of std:: transform with a parallel 
execution policy, this would exactly be the case. Unfortunately, this is not the 
right strategy for this problem, because every single point in the Mandelbrot 

/V ~ 

set shows A a very individual number of iterations. 

/V 

Our approach here is to make every single vector itemA which represents an 
individually printed character cell on the terminal later an asynchronously 
calculated future value. As source and target vector are w * h items large, 
which means 100 * 4 0 in our case, we have a vector of 4000 future values 
that are calculated asynchronously. If our system had 4000 CPU cores, then 

/V 

this would mean that we startA 4000 threads that do the Mandelbrot iteration 
really concurrently. On a normal system with fewer cores, the CPUs will just 
process one asynchronous item after the other per core. 

While the transform call with the asynchronized version 

/V 

ofA to iteration count itself does no calculation but setting up of threads 
and future objects, it returns practically immediately. The original version of 
the program blocked at this point because the iterations took so long. 

The parallelized version of the program does of course block somewhere , too. 
The function that prints all our values on the terminal must access the results 

/V 

from within the futures. In order to do that, it calls A x. get () on all the 
values. And this is the trick: while it waits for the first value to be printed, a 
lot of other values are calculated at the same time. So if the get () call of the 
first future returns, the next future might be ready for printing already too! 

In case w * h results in much larger numbers, there will be some measurable 
overhead in creating and synchronizing all these futures. In this case, the 
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overhead is not too significant. On myA laptop with an Intel i7 processor 
with 4 hyperthreading capable cores (which results in eight virtual cores), the 
parallel version of this program ran more than 3-5 times faster compared to 
the original program. The ideal parallelization would make it indeed 8 times 
faster. Of course, this speedup will vary between different computers, 



because it depends on a lot of factors. 



Implementing a tiny automatic 
parallelization library with std::future 


Most complex tasks can be broken down into subtasks. From all subtasks, we 
can draw an directedA acyclic graph (DAG) that describes which subtask 
depends on what other subtasks in order to finish the higher level task. Let us, 
for example, imagine that we want to produce the string "foo bar foo bar 

/V 

this that ", and we can only do this by creatingA single words and 
concatenate those with other words, or with themselves. Let's say this 
functionality is provided by three primitive functions create, concat, and 
twice. 


Taking this into account, we can draw the following DAG that visualizes the 
dependencies between them in order to get the final result: 



"foo bar foo bar this that" 


When implementing this in code, it is clear that everything can be 
implemented in a serial manner on one CPU core. Alternatively, all subtasks 













that depend on no other subtasks or other subtasks that already have been 
finished, can be executed concurrently on multiple CPU cores. 

It might perhaps seem tedious to write such code, even with std:: async 
because the dependencies between the subtasks need to be modeled. In this 
recipe, we will implement two little library helper functions that help to 
transform the normal functions create, concat, and twice to functions that 
work asynchronously. With those, we will find a really elegant way to set up 
the dependency graph. During execution, the graph will parallelize itself in a 
seemingly intelligent way in order to calculate the result as fast as possible. 

How to do it... 


In this section, we are going to implement some functions that simulate 
computation-intensive tasks that depend on each other, and let them run as 
parallel as possible: 

1. Let's first include all the necessary headers: 

#include <iostream> 

#include <iomanip> 

#include <thread> 

#include <string> 

#include <sstream> 

#include <future> 

using namespace std; 

using namespace chrono_literals; 

2. We need to synchronize concurrent access to cout, so let's use the 
synchronization helper from the other recipe in this chapter: 

struct pcout : public stringstream { 
static inline mutex cout mutex; 

~pcout() { 

lock_guard<mutex> 1 {cout_mutex}; 
cout << rdbuf(); 
cout.flush (); 


} 



3. Now let's implement three functions which transform strings. The first 
function shall create an std:: string object from a C-string. We let it 
sleep for 3 seconds to simulate that string creation is computation- 
heavy: 


static string create(const char *s) 

{ 

pcout{} << "3s CREATE " << quoted(s) << 1 \n 1 ; 
this_thread::sleep_for(3s); 
return {s}; 

} 

4. The next function accepts two string objects as arguments and returns 
their concatenation. We give it 5-second wait time to simulate that this is 
a time-consuming task: 


static string concat (const string &a, const string &b) 

{ 

pcout{} « "5s CONCAT " 

<< quoted(a) << " " 

<< quoted(b) << 1 \n 1 ; 
this_thread::sleep_for(5s); 
return a + b; 

} 

5. The last computation-heavy function accepts a string and concatenates it 
with itself. It shall take 3 seconds to do this: 

static string twice(const string &s) 

{ 

pcout{} << "3s TWICE " << quoted(s) << 1 \n'; 
this_thread::sleep_for(3s); 
return s + s; 

} 

/v 

6. We could now already use those functionsA in a serial program, but we 
want to get some elegant automatic parallelization. So let's implement 
some helpers for this. Attention please, the following three functions 
look really complicated, asynchronize accepts a function f and returns 
a callable object that captures it. We can call this callable object with 
any number of arguments, and then it will capture those together with f 
in another callable object which it returns to us. This last callable object 



can be called without arguments. It does then call f asynchronously with 

/\ 

all the arguments A it captures: 


template <typename F> 
static auto asynchronize(F f) 

{ 

return [f] (auto ... xs) { 
return [ = ] () { 

return async(launch::async, f, xs...); 

} ; 

}; 

} 

7. The next function will be used by the function we declare in the next 
step afterward. It accepts a function f, and captures it in a callable object 
that it returns. That object can be called with a number of future objects. 
It will then call . get () on all the futures, apply f to them and return its 

/V 

result:A 


template ctypename F> 
static auto fut_unwrap(F f) 

{ 

return [f] (auto ... xs) { 
return f (xs . get ()...); 

}; 

} 

8. The last helper function does also accept a function f. It returns a 
callable object that captures f. That callable object can be called with 
any number of callable objects as arguments, which it returns captured 
together with f in another callable object. That final callable object can 
then be called without arguments. It does then call all the callable 
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objects that A are captured in the xs ... pack. These return futures which 
need to be unwrapped with fut unwrap. The future-unwrapping and 
actual application of the real function f on the real values from the 
futures does again happen asynchronously using std:: async: 


template <typename F> 
static auto async_adapter(F f) 

{ 

return [f](auto ... xs) { 
return [ = ] () { 

return async(launch::async. 



fut_unwrap(f), xs ()...); 


}; 

}; 

} 

9. Ok, that was maybe kind of a crazy ride that was slightly reminiscent of 
the movie " Inception " because of the lambda expressions that return 
lambda expressions. We will have a very detailed look at this voodoo- 
code later. Now let's take the functions create, concat, and twice and 
make them asynchronous. The function async adapter makes a 
completely normal function wait for future arguments and return a 
future result. It is kind of a translating wrapper from the synchronous to 

/V 

the asynchronous world. We apply it to concat andA twice. We must 

/V 

useA asynchronize on create because it shall return a future, but we 
will feed it with real values instead of futures. The task dependency 
chain must begin with create calls: 

int main() 

{ 

auto pcreate (asynchronize(create)); 

auto pconcat (async_adapter(concat)); 

auto ptwice (async_adapter(twice)); 

/v 

10. Now we have automatically parallelizing functionsA that have the same 
names as the original synchronous ones, but with a p-prefix. Let us now 
set up a complex example dependency tree. First, we create the strings 
"too " and "bar ", which we immediately concatenate to "too bar ". 
This string is then concatenated with itself using twice. Then we create 
the strings "this " and "that ", which we concatenate to "this that 
". Finally, we concatenate the results to "too bar too bar this that 
". The result shall be saved in the variable callable. Then finallyA call 
callable (). get () in order to start the computation and wait for its 
return value, in order to also print that. No computation is done before 
we call callable o , and after this call, all the magic starts: 

auto result ( 
pconcat( 

ptwice( 

pconcat( 

pcreate("too " ) , 
pcreate("bar "))), 



pconcat( 

pcreate("this "), 
pcreate("that ")))); 

cout << "Setup done. Nothing executed yet.\n"; 
cout << result().get() << '\n'; 

} 

11. Compiling and running the program shows that all the create calls are 
performed at the same time, and then the other calls are performed. It 
looks as if they were scheduled intelligently. The whole program runs 
for 16 seconds. If the steps were not performed in parallel, it would take 
30 seconds to complete. Note that we need a system with at least four 
CPU cores to be able to perform all create calls at the same time. If the 
system had fewer CPU cores, then some calls would have to share CPUs 

/V 

which would of course then consumeA more time: 

$ ./chains 

Setup done. Nothing executed yet. 

3s CREATE "too " 

3s CREATE "bar " 

3s CREATE "this " 

3s CREATE "that " 

5s CONCAT "this " "that " 

5s CONCAT "too " "bar " 

3s TWICE"foo bar " 

5s CONCAT "too bar too bar " "this that " 
too bar foo bar this that 


How it works... 


A plain serial version of this program without any async and future magic 
would look like the following: 

int main() 

{ 

string result { 
concat ( 

twice( 

concat( 

create("foo "), 
create("bar "))), 



} 


concat ( 

create("this "), 
create("that "))) }; 

cout << result << '\n'; 


/\ /\ 

In this recipe, we wroteA the helper functionsA async adapter and 
asynchronize that helped US create new functions from create, concat, and 
twice. We called those new asynchronous functions pcreate, pconcat, and 
ptwice. 


Let us first ignore the complexity of the implementation of async adapter 
and asynchronize, in order to first have a look what this got us. 

The serial version looks similar to this code: 

string result {concat ( ... )}; 
cout << result << '\n'; 

The parallelized version looks similar to A the following: 

auto result (pconcat( ... ) ) ; 
cout << result 0 .get() << '\n'; 

Okay, now we get at the complicated part. The type of the parallelized result 
is not string, but a callable object that returns a future<string> on which 
we can call get (). This might indeed look crazy at first. 

So, how and why did we exactly end up with callable objects that return 
futures? The problem with our create, concat, and twice methods is, that 
they are slow, (okay, we made them artificially slow, because we tried to 
model real life tasks that consume a lot of CPU time). But we identified that 
the dependency tree which describes the data flow has independent parts that 
could be executed in parallel. Let's have a look at two example schedules: 



Single Core 


Multi Core 


time Core 0 time Core 0 Core 1 Core 2 Core 3 



/V 

On the left side, we seeA a single core schedule. All the function calls have 

/V _ 

to be done one after eachA other because we have only a single CPU. That 
means, that when create costs 3 seconds, concat costs 5 seconds and twice 
costs 3 seconds, it will take 30 seconds to get the end result. 

On the right side, we see a parallel schedule where as much is done in 
parallel as the dependencies between the function calls allow. In an ideal 
world with four cores, we can create all substrings at the same time, then 
concatenate them and so on. The m ini mal time to get the result with an 
optimal parallel schedule is 16 seconds. We cannot go faster if we cannot 

make the function calls themselves faster. With just four CPU cores we can 
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achieve A this execution time. We measurably achieved the optimal schedule. 
How did it work? 


We could naively write the following code: 


auto 

a 

(async 

(launch::async. 

create. 

"too ")); 


auto 

b 

(async 

(launch::async. 

create. 

"bar " ) ) ; 


auto 

c 

(async 

(launch::async. 

create. 

"this ")); 


auto 

d 

(async 

(launch::async. 

create. 

"that ")) ; 


auto 

e 

(async 

(launch::async. 

concat. 

a.get(), b.get ( 

))) ; 

auto 

f 

(async 

(launch::async. 

concat. 

c.get(), d.get ( 

))) ; 

auto 

g 

(async 

(launch::async. 

twice. 

e .get () ) ) ; 


auto 

h 

(async 

(launch::async. 

concat. 

g.get(), f.get( 

))) ; 


This is a good start for a, b, c, and d, which represent the four substrings to 








begin with. These are created asynchronouslyA in the background. 
Unfortunately, this code blocks on the line where we initialize e. In order to 
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concatenate a and b, we need to call get o A on both of them, which blocks 
until these values are ready. Obviously, this is not a good idea, because the 
parallelization stops being parallel on the first get () call. We need a better 
strategy. 

Okay, so let us roll out the complicated helper functions we wrote. The first 

One IS asynchronize: 


template <typename F> 
static auto asynchronize(F f) 

{ 

return [f](auto ... xs) { 
return [ = ] () { 

return async(launch::async, f, xs . . .); 

} ; 

}; 

} 

When we have a function int f(int, int) then we can do the following: 

auto f2 ( asynchronize(f) ); 

auto f3 ( f2(1, 2) ); 

auto f4 ( f3 () ) ; 

int result { f4.get() }; 

f 2 is our asynchronous version of f . It can be called with the same arguments 

like f, because it mimics f . Then it returns a callable object, which we save in 
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f 3. f 3 now captures f and the arguments A l, 2 , but it did not call anything 
yet. This is just about the capturing. 

When we call f 3 () now, then we finally get a future, because f 3 () does the 

async (launch: : async, f, l, 2); call! In that sense, the semantic meaning 

/\ /\ - 

ofA f 3 A is " Take the captured function and the arguments, and throw them 
together into std: :async.'\ 

The inner lambda expression that does not accept any arguments gives us an 

/V 

indirection. A With it, we can set up work for parallel dispatch but do not 

/V 

have to callA anything that blocks, yet. We follow the same principle in the 
much more complicated function async adapter: 



template ctypename F> 
static auto async_adapter(F f) 

{ 

return [f](auto ... xs) { 
return [ = ] () { 

return async(launch::async, fut_unwrap(f), xs ()...); 

} ; 

}; 

} 

This function does also first return a function thatA mimics f because it 
accepts the same arguments. Then that function returns a callable object that 
again accepts no arguments. And then that callable object finally differs from 
the other helper function. 


What does the async (launch:: async, fut_unwrap (f) , xs ()...); line 
mean? The xs () ... part means, that all arguments that are saved in pack xs 
are assumed to be callable objects (like the ones we are creating all the 
time!), and so they are all called without arguments. Those callable objects 
that we are producing all the time themselvesA produce future values, on 
which we can call get () . This is where fut unwrap comes into play: 

template <typename F> 
static auto fut_unwrap(F f) 

{ 

return [f](auto ... xs) { 
return f(xs. get () . . .) ; 

}; 

} 

fut unwrap just transforms a function f into a function object that accepts a 
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range of arguments. This function object does then callA . get () on all of 
them and then finally forwards them to f . 

Take your time to digest all of this. When we used this in our main function, 
then the auto result (pconcat (...)); call chain did just construct a 
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largeA callable object that contains all functions and all arguments. No async 
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call wasA done at this point yet. Then, when we called result o, we 
unleashed a little avalanche of async and . get () calls that come just in the 
right order to not block each other. In fact, no get () call happens before not 
all async calls have been dispatched. 



In the end, we can finally call . get () on the future value that result () 
returned, and there we have our final string. 



Chapter 10. Filesystem 


In this chapter, we will cover the following recipes: 

• Implementing a path normalizer 

/\ 

• Getting canonicalA file paths from relative paths 

• Listing all files in directories 

• Implementing a grep-like text search tool 

• Implementing an automatic file renamer 

• Implementing a disk usage counter 

• Calculating statistics about file types 

/\ 

• Implementing a tool that reduces folder sizeA by substituting duplicates 
with symlinks 



Introduction 


Working with filesystem paths is always tedious if we don't have a library 
that helps us because there are many conditions that we need to handle. 

Some paths are absolute, some are relative, and maybe they are not even 

/V /V 

straightforward because theyA also contain . A (current directory) and 

A _ 

. .A (parent directory) indirections. Then, at the same time, different 
operating systems use the slash / to separate directories (Linux, MacOS, and 
different UNIX derivatives), or the backslash \ (Windows). And of course 
there are different types of files. 

Since every other program that handles filesystem-related things needs such 

/V 

functionality, it is great to have the new filesystem libraryA in the C++17 

/V 

STL. The best thing about it A is that it works the same way for different 
operating systems, so we don't have to write different code for versions of our 

/V 

programs that support different operating systems.A 

/V 

In this chapter, we will firstA see how the path class works, because it is 
most central to anything else in this library. Then, we will see how powerful 
but yet simple to use directory iterator and 

recursive directory iterator classes are, while we do useful things with 

/\ /\ 

files. In the end, we will use A some smallA and simple example tools that do 
some real-life tasks related to the filesystem. From this point, it will be easy 
to build more complex tools. 



Implementing a path normalizer 


We start this chapter with a very simple example around the 

std:: filesystem: :path class and a helper function that intelligently 

normalizes filesystem paths. 

The result of this recipe is a little application that takes any filesystem path 
and returns us the same path in normalized form. Normalized means that we 

/V 

get an absolute path that contains no . or . .A path indirections. 

While implementing that, we will also see what details we need to pay 
attention to when working with this basic part of the filesystem library. 

How to do it... 


/V 

In this section, we willA implement a program that just accepts a filesystem 
path as a command-line argument and then prints it in normalized form. 

1. Includes come first, and then we declare that we use namespace std and 

filesystem. 

#include <iostream> 

#include <filesystem> 

using namespace std; 
using namespace filesystem; 


/v 

2. In the main function, we check whether A the user provided a command¬ 
line argument. If that is not the case, we error out and print how to use 
the program. If a path was provided, we instantiate a filesystem: :path 
object from it. 

int main(int argc, char *argv[]) 

{ 

if (argc != 2) { 

cout << "Usage: " << argv[0] << " <path>\n"; 




return 1; 

} 

const path dir {argv[l]}; 


3. Since we can instantiate path objects from any string, we cannot be sure 
if the path really exists on the filesystem of the computer. In order to do 
that, we can use the filesystem:: exists function. If it doesn't, we 
simply error out again. 

if ( !exists(dir)) { 

cout << "Path " << dir << " does not exist.\n"; 
return 1; 

} 

4. Okay, at this point, we are pretty sure that the user provided some 

/V 

existing path knowing thatA we can ask for a normalized version of it, 
which we then print, filesystem: : canonical returns US another path 

/v 

object. We could print it directly, but the pathA type overload of the « 
operator surrounds paths with quotation marks. In order to avoid that, 
we can print a path through its . c_str() or . string o method. 

cout << canonical(dir).c_str() << 1 \n 1 ; 

} 

5. Let's compile the program and play with it. When we execute it in my 
home directory on the relative path "src", it will print the full absolute 
path. 

$ ./normalizer src 
/Users/tfc/src 


6. When we run the program in my home directory again, but give it a 
quirky relative path description that first enters my Desktop folder, then 
steps out of it again using .., then enters the Documents folder and steps 
out again in order to finally enter the src directory, the program prints 
the same path as before! 

$ ./normalizer Desktop/../Documents/../src 
/Users/tfc/src 



How it works... 


/V 

As a starter on std:: filesystem, this recipe isA still fairly short and 
straightforward. We initialized a path object from a string that contains a 

_ /V 

filesystem path description. The std:: filesystem: :pathA class plays a very 

/v 

central role whenever we use the filesystem libraryA because most of the 
functions and classes relate to it. 

/V /V 

Using theA filesystem: :existsA function, we were able to check if the 
path really exists. Up to that point, we could not be sure about that, because it 
is indeed possible to create path objects that do not relate to an existing 
filesystem object, exists just accepts a path instance and returns true if it 
really exists. The function is already able to determine itself if we gave it an 
absolute or a relative path, which makes it very comfortable to use. 

Finally, we used filesystem:: canonical on the directory in order to print it 
in normalized form. 

path canonical(const paths p, const paths base = current_path()); 
canonical accepts a path and as an optional second argument, it accepts 

/V /V 

another path. The second path base is prepended to path pA if p is A a relative 
path. After doing that, canonical tries to remove any . and .. path 
indirections. 

While printing, we used the . c str () method on the canonicalized path. The 
reason for this is that the overload of operator« for output streams 
surrounds paths with quotation marks, which we mayA not always want. 

There’s more... 

/\ 

canonical throws a f iiesystem error type exceptionA if the path we want 
to canonicalize does not exist. In order to prevent that, we checked our 
filesystem path with exists. But was that check really sufficient to avoid 
getting unhandled exceptions? No. 

Both exists and canonical can throw badaiioc exceptions. If those hit US, 



one could argue that the program is doomed anyway. A far more critical, and 
also much more probable problem would occur if, between us checking if the 
file exists and canonicalizing it, someone else renames or deletes the 
underlying file! In that case, canonical would throw a filesystem_error, 
although we checked for the file's existence before. 

/V 

Most filesystem functions have an additionalA overload that takes the same 
arguments, but also an std: :error_code reference. 

path canonical(const paths p, const paths base = current_path()); 
path canonical(const paths p, error_codeS ec); 
path canonical(const std::filesystem::paths p, 

const std::filesystem::paths base, 
std::error codes ec ); 


This way we can choose if we surround our filesystem function calls with 
try-catch constructs or check the errors manually. Note that this only 
changes the behavior of filesystem-related errors! With and without the ec 

/V 

parameter, more fundamental exceptions, for example,A bad aiioc, can still 
be thrown if the system runs out of memory. 



/V 

Getting canonicalA file paths from relative 
paths 


In the last recipe, we already canonicalized/normalized paths. The 
filesystem: :path class is, of course, capable of more things than just 
holding and checking paths. It also helps us in composing paths from strings 
easily, and also to decompose them again. 

At this point, path does already abstract operating system details away from 

/V 

us, but there are also certain instancesA where we still need to keep such 
details in mind. 

We will see how to deal with paths and their composition/decomposition by 
playing around with absolute and relative paths. 

How to do it... 

/V 

In this section, we will A play around with absolute and relative paths in order 

/V 

to see the strengths of the path class and the helper functions around it. A 

1. First, we include all the necessary headers and declare that we use 
namespace std and sfiiesystem. 

#include <iostream> 

#include <filesystem> 

using namespace std; 
using namespace filesystem; 


2. Then, we declare an example path. At this point, it is not important that 
the text file it refers to really exists. There are some functions, however, 
that throw exceptions if the underlying file does not exist. 


int main() 
{ 




path p {"testdir/foobar.txt"}; 

/v 

3. We will have a look at fourA different filesystem library functions now. 
current path returns us the path the program is currently executed in, 
the working directory, absolute accepts a relative path like our path p 
and returns the absolute, nonambiguous path in the whole filesystem. 

/V 

system compieteA does practically the same as absolute on Linux, 
MacOS, or UNIX-like operating systems. On Windows, we would get 
the absolute path additionally prepended by the disk volume letter (for 

/V 

example,A "C: "). canonical does again the same as absolute does, but 

/v 

then additionally removes any ". "A (short for "this directory") or 

/V 

".. "A (short for "one directory up") indirections. We will play with 

/V 

such indirections in the folio wing A steps: 

cout << "current_path 

<< "\nabsolute_path 
<< "\nsystem_complete 
<< "\ncanonical(p) 

« ' \ n ' ; 

/v 

4. Another nice thing about the path class is that it overloads theA / 

/V _ 

A operator. This way we can concatenate folder names and filenames 
using / and compose paths from that. Let's try it out and print a 
composed path. 

cout << path{"testdir"} / "foobar.txt" << '\n'; 

5. Let's play with canonical and composed paths. By giving canonical a 
relative path such asA "foobar. txt" and a composed absolute path 
current path () / "testdir", it should return us the existing absolute 
path. In another call, we give it our path p (which 

/V 

isA "testdir/foobar. txt") and provide it an absolute path that is 
current path o, whichA directs us into "testdir" and up again. This 
should be the same as current path (), because of the indirection. In 
both calls, canonical should return us the same absolute path. 

cout << "canonical testdir : " 

<< canonical("foobar.txt", 

current_path() / "testdir") 

<< "\ncanonical testdir 2 : 


" << current_path() 

" << absolute(p) 

" << system_complete(p) 
" << canonical(p) 


fV 




<< canonical (p, current_path() / "testdir/..") 

« ' \ n ' ; 


6. We can also test for the equivalence of two paths that are not canonical, 
equivalence canonicalizes the paths, which it accepts as arguments and 
returns true if they describe the same path after all. For this test, the 

/\ /V 

path must really exist, A otherwise,A it throws an exception. 


cout << 
<< 

<< 

} 


"equivalence: " 

equivalent("testdir/foobar.txt", 

"testdir/../testdir/foobar.txt") 

' \ n ' ; 


7. Compiling and running the program yields the following output. 

/V 

current path () returns the home folder on my laptopA because I 

executed the application from there. Our relative path p has been 

prepended with this directory by absoiute path, system compiete, and 

canonical. We see that absolute_path and system_complete yield 

/\ 

exactly the same path on my systemA because it is a Mac (it would be 
the same on Linux). On a Windows machine, system compiete would 
have prepended "c:or whatever drive the working directory is located 
in. 


$ ./canonical_filepath 
current_path: "/Users/tfc" 

absolute_path : "/Users/tfc/testdir/foobar.txt" 
system_complete : "/Users/tfc/testdir/foobar.txt" 
canonical(p): "/Users/tfc/testdir/foobar.txt" 

"testdir/foobar.txt" 

canonical testdir : "/Users/tfc/testdir/foobar.txt" 
canonical testdir 2 : "/Users/tfc/testdir/foobar.txt" 
equivalence: 1 

8. We do not handle any exceptions in our short program. If we remove 

/\ /\ /\ /\ 

theA foobar. txtA file in theA testdirA directory, then the program 
aborts its execution due to an exception. The canonical function 
requires the path to exist. There is also aA weakiy canonicaiA function 

/v 

that does not come with this requirement.A 

$ ./canonial_filepath 
current path: "/Users/tfc" 



absolute_path : "/Users/tfc/testdir/foobar.txt" 
system_complete : "/Users/tfc/testdir/foobar.txt" 

terminate called after throwing an instance of 

'std::filesystem::vl::_cxxll::filesystem_error' 

what():filesystem error: cannot canonicalize: 

No such file or directory [testdir/foobar.txt] [/Users/tf 


How it works... 


The goal of this recipe is to see how easy it is to compose new paths on the 

_ /V /\ 

fly. This is mainly because theA path class has a handy overload for theA / 

/\ /\ 

A operator. In addition to that, the filesystem functions get along wellA with 
relative and absolute paths, as well as with paths that contain . and .. 
indirections. 

There is quite a jungle of functions that return parts of a path instance, with 
or without transformations. We are not going to list all functions there are 
hereA because a short glance into the C++ reference is the best way to get an 
oversight. 

The member functions of the path class, however, might be worth a closer 

look. Let's see which part of a path is returned by what member function of 

/\ _ 

path. A The following diagram also shows how Windows paths are slightly 
different from UNIX/Linux paths. 


Windows: 

UNIX/Linux: 


rootjpath() 

relative _path() 


parent_path() 

Users\foo\ 

Users/foo/ 

filenamef) 

C: 

\ 

/ 

stem() 

filename 

filename 

extensionf) 

.txt 

.txt 

' 

l _ 


i root_directory() 
root_name() 


You can see that the diagram shows what the member functions 

A /V 

ofA pathA return for an absolute path. For relative paths, root path, 
root name, and root directory are empty, relative path then just returns 




/\ 

the path ifA it is relative already. 



Listing all files in directories 


/V 

Of course, every operating system that offers filesystem supportA also comes 
with some kind of utility that does just list all files within a directory in the 
filesystem. The simplest examples are the is command on Linux, MacOS, 
and other UNIX-related operating systems. In DOS and Windows, there is 
the dir command. Both list all files in a directory and provide supplemental 
information such asA file size, permissions, and so on. 

/\ 

Reimplementing such a toolA is, however, also a nice standard task to get 
going with directoiy and file traversal. So, let's just do that! 

Our own 1 s/dir utility will be able to list all items in a directory by name, 
indicate what kind of items there are, list their access permission flags, and 

/V 

displayA the number of bytes they occupy on the filesystem. 

How to do it... 


/\ 

In this section, we willA implement a little tool that lists all files in any user 
provided directory. It will not only list the filenames, but also their type, size, 
and access permissions. 

1. First, we need to include some headers and declare that we use the 

/V 

namespacesA std and filesystem by default. 

#include <iostream> 

#include <sstream> 

#include <iomanip> 

#include <numeric> 

#include <algorithm> 

#include <vector> 

#include <filesystem> 


using namespace std; 
using namespace filesystem; 




2. One helper function that we are going to need is fiie_info. It accepts a 

/V 

directory entry object referenceA and extracts the path from it, as 
well as a fiie status object (using the status function), which 
contains file type and permission information. Finally, it also extracts 
the size of the entry if it is a regular file. For directories or other special 
files, we plainly return a size of o. All this information is bundled into a 
tuple. 


static tuple<path, file_status, size_t> 

file_info(const directory_entry Sentry) 

{ 

const auto fs (status(entry)); 
return {entry.path(), 
fs, 

is_regular_file(fs) ? file_size(entry.path()) 

} 

/v /v 

3. Another helper function that we willA need is type char. AA path 
cannot only represent directories and simple text/binary files. Operating 
systems provide a variety of other types that abstract something else, 
such asA hardware device interfaces in the form of so-called 
character/block files. The STL filesystem library provides a lot of 
predicate functions for them. This way we can return the letter ' d' for 
directories, the letter ' f' for regular files, and so on. 


static char type_char(file_status fs) 
{ 


if 


(is_ 

directory(fs)) 

{ return 

'd' ; 

} 

else 

if 

(is_ 

symlink(fs)) 

{ return 

'1' ; 

} 

else 

if 

(is_ 

character file(fs)) 

{ return 

' c' ; 

} 

else 

if 

(is_ 

block file(fs)) 

{ return 

'b' ; 

} 

else 

if 

(is_ 

fifo (fs)) 

{ return 

'p'; 

} 

else 

if 

(is_ 

socket(fs)) 

{ return 

's'; 

} 

else 

if 

(is_ 

other(fs)) 

{ return 

' o' ; 

} 

else 

if 

(is_ 

regular file(fs)) 

{ return 

' f'; 

} 


return '?'; 

} 

/v /v 

4. Yet another helper we will A need is the A rwxA function. It accepts a 
perms variable (which is just an enum class type from the filesystem 

/V 

library) and returns a string such asA " rwxrwxrwx " that describes the 




file's permission settings. The first group of "rwx" characters describes 
the read, write, and execution permissions for the owner of the file. The 
next group describes the same rights for all users that are part of the user 
group the file belongs to. The last character group describes which 
rights everyone else has for accessing the file. A string such 

/V 

asA " rwxrwxrwx " means that everyone can access the object in any way. 
"rw-r—r—" means that only the owner can read and modify the file, 
while anyone else can only read it. We just compose a string from such 
read/write/execute character values, permission bit by permission bit. A 
lambda expression helps us with the repetitive work of checking if the 
perms variable p contains a specific owner bit and then returns 1 - 1 or 
the right character. 

static string rwx(perms p) 

{ 

auto check ([p](perms bit, char c) { 

return (p & bit) == perms::none ? 1 : c; 

}) ; 

return {check(perms::owner_read, 'r'), 
check(perms::owner_write, 'w'), 
check(perms::owner_exec, 'x'), 
check(perms::group_read, 'r'), 
check(perms::group_write, 'w'), 
check(perms::group_exec, 'x'), 
check(perms::others_read, 'r'), 
check(perms::others_write, 'w'), 
check(perms::others_exec, 'x')}; 

} 

5. Finally, the last helper function accepts an integral file size and converts 
it to a better to read form. We just ignore the period while dividing 
numbers down and floor them to the nearest kilo, mega, or giga 
boundary. 

static string size_string(size_t size) 

{ 

stringstream ss; 

if (size >= 1000000000) { 

ss << (size / 1000000000) << 'G'; 

} else if (size >= 1000000) { 

ss << (size / 1000000) << 'M'; 

} else if (size >= 1000) { 



} 


ss << (size / 1000) << 'K'; 
} else { ss << size << 'B'; } 

return ss . str (); 


6. Now we can finally implement the main function. We begin with 
checking if the user provided a path in the command line. If he didn't, 
we just take the current directory ".Then, we check if the directory 
exists. If it doesn't, we can't possibly list any files. 

int main(int argc, char *argv[]) 

{ 

path dir {argc > 1 ? argvfl] : 
if ( !exists(dir)) { 

cout << "Path " << dir << " does not exist.\n"; 
return 1; 

} 

/v /v 

7. Now, we willA fill a vectorA with file information tuples just like our 

/V 

first helper function A file_info returns from directory_entry 

/V 

objects. A We instantiate a directory iterator and give its 
constructor the path object, which we created in the last step. While 
iterating with the directory iterator, we transform the directory entry 
objects to file information tuples and insert them into the vector. 

vector<tuple<path, file_status, size_t>> items; 

transform(directory_iterator{dir}, {}, 

back_inserter(items), file_info); 

/v 

8. Now we have all information saved in the vectorA items and can simply 
print it using all the helper functions we wrote. 


for (const auto &[path, status, size] : items) { 
cout << type_char(status) 

<< rwx(status.permissions()) << " " 

<< setw(4) << right << size_string(size) 
<< " " << path.filename().c_str() 

« ’\n’; 


} 


} 



9. Compiling and running the project with a file path in the offline version 

of the C++ documentation yields the following output. We see that the 

/\ 

folder only contains directories and plain filesA because there are only 
' d' and ' f' entries as first characters of all output lines. These files 
have different access permissions, and of course different sizes. Note 
that the files appear in alphabetical order of their names, but we cannot 

/V 

really rely on thatA because alphabetic ordering is not required by the 
C++17 standard. 

$ ./list ~/Documents/cpp_reference/en/cpp 

drwxrwxr-xOBalgorithm 

frw-r—r— 8 8Kalgorithm.html 

drwxrwxr-xOBatomic 

frw-r—r— 35Katomic.html 

drwxrwxr-xOBchrono 

frw-r—r— 34Kchrono.html 

frw-r—r— 21Kcomment.html 

frw-r —r— 21Kcomments.html 

frw-r--r--220Kcompiler_support.html 

drwxrwxr-xOBconcept 

frw-r—r— 67Kconcept.html 

drwxr-xr-xOBcontainer 

frw-r--r--2 8 5Kcontainer.html 

drwxrwxr-xOBerror 

frw-r—r— 52Kerror.html 


How it works... 


In this recipe, we iterated over files, and for every file, we checked its status 
and size. While all our per-file operations are fairly straightforward and 
simple, our actual directory traversal looked a bit magic. 

In order to traverse our directory, we just instantiated a directory iterator 
and then iterated over it. Traversing a directory is fantastically simple with 
the filesystem library. 


for (const directory_entry &e : directory_iterator{dir}) { 

// do something 

} 


There is not much more to say about this class apart from the following 



things: 


• It visits every element of the directory once 

• The order in which the directory elements are iterated is unspecified 

• Directory elements . and .. are already filtered out 

However, it might be noticeable that directory iterator seems to be an 
iterator, and an iter able range at the same time. Why? In the minimal for 
loop example we just had a look at, it was used as an iterable range. In the 
actual recipe code, we used it like an iterator: 

transform(directory_iterator{dir}, {}, 

back_inserter(items), file_info); 

The truth is, it is just an iterator class type, but the std:: begin and std: :end 

_ /\ 

functions provide overloads for this type. This way we can call theA begin 
/\ 

and endA function on this kind of iterator and they return us iterators again. 
That might lookA strange at first sight, but it makes this class more useful. 



Implementing a grep-like text search tool 


Most operating systems come equipped with some kind of local search 
engine. Users can fire it up with some keyboard shortcut and then just enter 
what local file they are looking for. 

Before such features came up, command-line users already searched through 

/V /V 

files with tools such asA grepA or awk. The user can simply type "grep -r 
f oobar ." and the tool will crawl recursively through the current directory 

/V /\ 

and find any file that contains the A "f oobar "A string. 

/V 

In this recipe, we will implement exactly such anA application. Our little grep 

/V 

clone will just accept a pattern from theA command line, and then recursively 
search through the directory we are in at the time of the application start. It 
will then print the name of every file that matches our pattern. The pattern 
matching will be applied linewise, so we can also print on which exact line 
numbers a file is matching the pattern. 

How to do it... 


/\ 

We willA implement a little tool that searches for user-provided text patterns 

_ /V 

in files. The tool works similarA to the UNIX tool grep, but will not be as 

/V 

mature and powerful, for the sake of simplicity.A 

1. First, we need to include all the necessary headers and declare that we 
use namespace std and filesystem. 

#include <iostream> 

#include <fstream> 

#include <regex> 

#include <vector> 

#include <string> 

#include <filesystem> 


using namespace std; 
using namespace filesystem; 





/V 

2. We implement a helper function first. It accepts a file pathA and a 
regular expression object that describes the pattern we are looking for. 

/V 

Then, we instantiate a vectorA that shall contain pairs of matching line 
numbers and their content. And we instantiate an input file stream object 

/V 

from which we will A read and pattern-match the content, line by line. 

static vector<pair<size_t, string>> 
matches(const path &p, const regex &re) 

{ 

vector<pair<size_t, string>> d; 
ifstream is {p.c_str()}; 

3. We traverse the file line by line using the getiine function, 
regexsearch returns true if the string contains our pattern. If this is 
the case, then we put the line number and the string into the vector. 
Finally, we return all collected matches. 

string s; 

for (size_t line {1}; getiine(is, s); ++line) { 
if (regex_search(begin(s), end(s), re)) { 
d.emplace_back(line, move(s)); 

} 

} 

return d; 

} 

/v 

4. In the main function, we first check whetherA the user provided a 
command-line argument that we can use as the pattern. If not, we error 
out. 

int main(int argc, char *argv[]) 

{ 

if (argc != 2) { 

cout << "Usage: " << argv[0] << " <pattern>\n"; 
return 1; 

} 

5. Next, we construct a regular expression object from the input pattern. If 
the pattern is not a valid regular expression, this would lead to an 
exception. If such an exception occurs, we catch it and error out. 


regex pattern; 



try { pattern = regex{argv[1]}; } 

catch (const regex_error &e) { 

cout << "Invalid regular expression provided.n"; 
return 1; 

} 

6. Now, we can finally iterate over the filesystem and look for pattern 
matches. We use recursive_directory_iterator to iterate over all the 
files in the working directory. It works exactly like 
directory iterator in the previous recipe, but it also descends down 
into subdirectories. This way we don't have to manage recursion. On 
every entry, we call our helper function matches. 


for (const auto Sentry : 

recursive_directory_iterator{current_path()}) { 

auto ms (matches(entry.path(), pattern)); 


7. For every match (if any) we print the file path, its line number, and the 
matching line's complete content. 


} 


for (const auto &[number, content] : ms) { 

cout << entry.path().c_str() << << number 

<< " - " << content << 1 \n 1 ; 

} 


8. Let's prepare a file called "foobar. txt", which contains some test lines 
we can search for. 


f oo 
bar 
baz 


9. Compiling and running yields the following output. I launched the app 
in the /users/tfc/testdir folder on my laptop, first with the pattern 
"bar". Within that directory, it found the second line of our foobar. txt 
fileA and another file "texti. txt" that is located in testdir/diri. 

$ ./grepper bar 

/Users/tfc/testdir/dirl/textl.txt:1 - foo bar bla blubb 

/Users/tfc/testdir/foobar.txt:2 - bar 



10. Launching the app again, but this time with the pattern "baz", it finds 
the third line of our example text file. 

$ ./grepper baz 

/Users/tfc/testdir/foobar.txt:3 - baz 

How it works... 


Setting up and using a regular expression in order to filter the content of files 

/V 

isA certainly the main task of this recipe. However, let's concentrate on 

/V 

recursive directory iteratorA because filtering recursively iterated files 
was just our motivation to use this special iterator class in this recipe. 


Just like directory_iterator, recursive_directory_iterator iterates 
over elements of a directory. Its specialty is to do this recursively, as its name 
tells. Whenever it hits a filesystem element that is a directory , it will yield a 
directory entry instance to this path, but then also descend down into it in 
order to iterate its children, too. 

recursive directory iterator has some interesting member functions: 

• depth (): This tells usA how many levels the iterator has currently 

/V 

descended down into subdirectories.A 

• recurs ionpending (): This tells us if the iterator is going to descend 
down after the element it currently points to. 

• disabie recursion pending (): This can be called to keep the iterator 

/\ 

from descending into the next subdirectoryA if it is currently pointing to 
a directory into which it would descend. This means that calling this 
method has no effect if we call it too early. 

• pop () : This aborts the current recursion level and goes one level up in 
the directory hierarchy to continue from there. 

There’s more... 

Another thing to know about is the directory options enum class. The 
constructor of recursive_directory_iterator does indeed accept a value 
of this type as a second argument. The default value which we have been 



implicitly using is directory options: :none. The other values are: 

• foiiow directory symiink: This allows the recursive iterator to follow 
symbolic l ink s to directories 

• skip permission denied: This tells the iterator to skip directories that 

/v 

would otherwise result in errorsA because permission to access is denied 
by the filesystem 

/V /v 

These options can be combined with theA | A operator. 



Implementing an automatic file renamer 


This recipe is motivated by a situation I find myself in pretty often. When 
collecting picture files from holidays, for example, from different friends and 
also different photo devices in one folder, the file endings often look 
different. Some JPEG files have a . jpg extension, some have .jpeg, and 
some others even have . jpeg. 

Some people might prefer to homogenize all extensions. It would be useful to 
rename all files with a single command. At the same time, we could remove 
spaces ' ' and substitute them by underscores ' ', for example. 

In this recipe, we will implement such a tool and call it renamer. It will 
accept a range of input patterns and their substitutes like this: 

$ renamer jpeg jpg JPEG jpg 

In that case, renamer will iterate recursively through the current directory and 
search for the patterns jpeg and jpeg in all filenames. It will substitute both 
with jpg. 


How to do it... 


/\ /V 

We willA implement a tool that recursively scansA all files within a directory 

and matches their filenames with patterns. All matches are replaced with user 

/\ 

provided tokens and the affected files are renamed accordingly.A 

1. First, we need to include a few headers and declare that we use 
namespaces std and filesystem. 

#include <iostream> 

#include <regex> 

#include <vector> 

#include <filesystem> 


using namespace std; 




using namespace filesystem; 


2. We implement a short helper function that accepts an input file path in 
the form of a string and a range of replacement pairs. Each replacement 
pair consists of a pattern and its replacement. While looping through the 
replacement range, we use regex repiace to feed it with the input 
string and let it return the transformed string. Afterward, we return the 
resulting string. 


template <typename T> 

static string replace(string s, const T Sreplacements) 

{ 

for (const auto &[pattern, repl] : replacements) { 
s = regex_replace(s, pattern, repl); 

} 

return s; 

} 

3. In the main function, we first validate the command line. We accept 

/V 

command-line arguments in pairs A because we want patterns together 
with their replacements. The first element of argvA is always the 
executable name. This means that if the user provides at least one pair or 
more, then argc must be odd and not smaller than 3. 

int main(int argc, char *argv[]) 

{ 

if (argc < 3 || argc % 2 != 1) { 

cout << "Usage: " << argv[0] 

<< " <pattern> <replacement> ...\n"; 
return 1; 

} 

/v 

4. Once we checked that there are pairs of input, we willA fill a 

/\ 

vectorA with these. 


vector<pair<regex, string>> patterns; 

for (int i {1}; i < argc; i += 2) { 

patterns.emplace_back(argv[i], argv[i + 1]); 

} 

5. Now we can iterate over the filesystem. For the sake of simplicity, we 



just define the application's current path as the directory to iterate 

/\ 

over.A For every directory entry, we extract its original path to 

/\ /\ _ 

theA opathA variable. Then, we take only the filename without the rest 

/\ 

of thisA path and transform it according to the list of patterns and 
replacements we collected before. We take a copy of opath, call it 

/V /V /V 

rpath, and replace itsA filename partA with theA new filename. 


for (const auto Sentry : 

recursive_directory_iterator{current_path()}) { 

path opath {entry.path()}; 

string rname {replace(opath.filename().string() , 

patterns)}; 


path rpath {opath}; 

rpath.replace_filename(rname); 


6. For all files that are affected by our patterns, we print that we rename 
them. In case the resulting filename from replacing the patterns does 
already exist, we can't proceed. Let's just skip such files. We could of 
course alternatively just append some number to the path or something 
else to resolve the name clash. 


if (opath != rpath) { 

cout << opath.c_str() << " --> " 

<< rpath.filename().c_str() << 1 \n 1 ; 
if (exists(rpath)) { 

cout << "Error: Can't rename." 

" Destination file exists.\n"; 

} else { 

rename(opath, rpath); 


7. Compiling and running the program in an example directory yields the 
following output. I have put some JPEG pictures into the directory but 
have given them different name endings jpg, jpeg, and jpeg. Then, I 
executed the program with the patterns jpeg and jpeg and chose jpg as 

A _ 

theA replacement for both. The result is a folder with homogenous 
filename extensions. 



$ Is 

birthday_party.jpeg holiday_in_dubai.jpgholiday_in_spain.jp 
trip_to_new_york.JPEG 
$ ../renamer jpeg jpg JPEG jpg 

/Users/tfc/pictures/birthday_party.jpeg --> birthday_party. 
/Users/tfc/pictures/trip_to_new_york.JPEG --> trip_to_new_y 
$ Is 

birthday_party.jpg holiday_in_dubai.jpg holiday_in_spain.jp 
trip to new york.jpg 



Implementing a disk usage counter 


We already implemented a tool that works like is on Linux/MacOS, or dir 
on Windows, but just as these tools, it doesn't print the file size for 
directories. 

In order to get the size equivalent of a directory, we would have to descend 
down into it and sum up the size of all files that it contains. 

/V 

In this recipe, we willA implement a tool that does just that. The tool can be 
run on any folder and will summarize the accumulated size of all directory 
entries. 


How to do it... 


/\ 

In this section, we willA implement an app that iterates over a directory and 
lists the file size of each entry. This is simple for regular files, but if we are 
looking at a directory entry that itself is a directory, then we have to look into 
it and summarize the size of all the files it holds. 

1. First, we need to include all the necessary headers and declare that we 
use namespace std and filesystem. 

#include <iostream> 

#include <sstream> 

#include <iomanip> 

#include <numeric> 

#include <filesystem> 

using namespace std; 
using namespace filesystem; 


2. Then we implement a helper function that accepts a directory entry as 
an argument and returns its size in the filesystem. If it is not a directory, 
we simply return the file size calculated by f iie size. 




static size_t entry_size(const directory_entry Sentry) 

{ 

if (!is_directory(entry)) { return file_size(entry); } 

3. If it is a directory, we need to iterate over all its entries and calculate 
their size. We end up calling our own entry size helper function 

/V 

recursivelyA if we stumble upon subdirectories again. 

return accumulate(directory_iterator{entry}, {}, Ou, 

[](size_t accum, const directory_entry &e) { 

return accum + entry_size(e); 

}) ; 

} 

4. For better readability, we use the same size string function as in other 
recipes in this chapter. It just divides large file sizes in to shorter and 
nicer ones to read strings with kilo, mega, or giga suffix. 

static string size_string(size_t size) 

{ 

stringstream ss; 

if (size >= 1000000000) { 

ss << (size / 1000000000) << 'G'; 

} else if (size >= 1000000) { 

ss << (size / 1000000) << 'M'; 

} else if (size >= 1000) { 

ss << (size / 1000) << 'K'; 

} else { ss << size << 'B'; } 

return ss.str (); 

} 

5. The first thing we need to do in the main function is to check 

/\ 

whether A the user provided a filesystem path on the command line. If 
that is not the case, we just take the current folder. Before proceeding, 

/V 

we check whether A it exists. 

int main(int argc, char *argv[]) 

{ 

path dir {argc > 1 ? argvfl] : 
if ( !exists(dir)) { 

cout << "Path " << dir << " does not exist.\n"; 
return 1; 



} 


6. Now, we can iterate over all directory entries and print their sizes and 
names. 


for (const auto Sentry : directory_iterator{dir}) { 

cout << setw(5) << right 

<< size_string(entry_size(entry)) 

<< " " << entry.path().filename().c_str() 

« '\n'; 

} 

} 

7. Compiling and running the program yields the following results. I 
launched it in a folder in the C++ offline reference. As it contains 
subfolders too, our recursive file size summary helper is immediately 
helpful. 

$ ./file_size ^/Documents/cpp_reference/en/ 

19M c 

12K c.html 

147M cpp 
17K cpp.html 
22K index.html 
22K Main_Page.html 


How it works... 


The whole program revolves around using f iie size on regular files. If the 
program sees a directory, it recursively descends down into it and calls 
fiie size on all its entries. 

The only thing we did to distinguish if we call f iiesize directly or if we 
need the recursion strategy was asking the is directory predicate. This 
works well for directories that only contain regular files and directories. 

As simple as our example program is, it would crash under the following 
conditions, because of unhandled exceptions: 

• fiie size only works on regular files and symbolic l ink s. It throws an 
exception in any other case. 



• Although f iie size works on symbolic links, it still throws an 
exception if we call it on a broken symbolic l ink . 

In order to make this example recipe program more mature, we need more 

/V 

defensive programming against the wrong type of filesA and handling of 
exceptions. 



Calculating statistics about file types 


In the last recipe, we implemented a tool that lists the size of all members of 
any directory. 

In this recipe, we will be counting sizes recursively, too, but this time we will 
accumulate the size of each file to their filename extension. This way we can 
print the user a table that lists how many files of each file type we have, and 
what the average size of such file types is. 

How to do it... 


/V 

In this section, we willA implement a little tool that recursively iterates over a 
given directory. While doing that, it counts the number and size of all files, 
grouped by their extensions. Finally, it prints which filename extensions exist 
within that directory, how many there are per extension, and their average file 
size. 

1. We need to include necessary headers and we declare that we use 
namespace std and filesystem. 

#include <iostream> 

#include <sstream> 

#include <iomanip> 

#include <map> 

#include <filesystem> 

using namespace std; 
using namespace filesystem; 


2. The size string function was already helpful in other recipes. It 
transforms file sizes to human-readable strings. 

static string size_string(size_t size) 

{ 

stringstream ss; 

if (size >= 1000000000) { 




ss << (size / 1000000000) << 'G'; 

} else if (size >= 1000000) { 

ss << (size / 1000000) << 'M'; 

} else if (size >= 1000) { 

ss << (size / 1000) << 'K'; 

} else { ss << size << 'B'; } 

return ss.str (); 

} 

3. Then, we implement a helper function that accepts a path object as its 
argument and iterates over all files within that path. On its way, it 
collects all information in a map that maps from filename extensions to 
pairs that contain the total number and accumulated size of all files that 
have the same extension. 

static map<string, pair<size_t, size_t>> ext_stats(const pa 
{ 

map<string, pair<size_t, size_t>> m; 

for (const auto Sentry : 

recursive_directory_iterator{dir}) { 

4. If a directory entry is a directory itself, we skip it. Skipping it at this 
point does not mean that we are not recursively descending into it. 
recursive directory iterator Still does that, but we do not want to 

/v 

look at the directory entries themselves.A 

const path p {entry.path()}; 

const file_status fs {status(p)}; 

if (is_directory(fs)) { continue; } 

5. Next, we extract the extension part of the directory entry string. If it has 
no extension, we simply skip it. 

const string ext {p.extension().string()}; 
if (ext.length() == 0) { continue; } 

6. Next, we calculate the size of the file we are looking at. Then, we look 

up the aggregate object in the map for this extension. If there are 

/\ 

yet A none at this point, it is created implicitly. We simply increment the 



file count and add the file size to the size accumulator. 


const size_t size {file_size(p)}; 

auto &[size_accum, count] = m[ext]; 

size accum += size; 
count += 1; 

} 

7. Afterward, we return the map. 

return m; 

} 

8. In the main function, we take either a user-provided path from the 

/V 

command lineA or the current directory. Of course, we need to check 

/V /\ 

whetherA it existsA because it would not make sense to continue 
otherwise. 


int main(int argc, char *argv[]) 

{ 

path dir {argc > 1 ? argvfl] : 
if ( !exists(dir)) { 

cout << "Path " << dir << " does not exist.\n"; 
return 1; 

} 

/v 

9. We can immediately iterate over the map that A ext stats gives us. 
Because the accum size items in the map contain the sum of all files 
with the same extension, we divide this sum by the total number of such 
files before printing it. 

for (const auto &[ext, stats] : ext_stats(dir)) { 

const auto &[accum_size, count] = stats; 

cout << setw(15) << left << ext << " 

<< setw(4) << right << count 
<< " items, avg size " 

<< setw(4) << size_string(accum_size / count) 
« ' \r.' ; 

} 

} 



10. Compiling and running the program yields the following output. I gave 
it a folder from the offline C++ reference as a command-line 
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argument. A 

$ ./file_type ~/Documents/cpp_reference/ 

.css :2 items, avg size41K 
.gif :7 items, avg size 902B 
.html: 4355 items, avg size38K 
.js:3 items, avg size 4K 
.php :1 items, avg size 739B 
.png : 34 items, avg size 2K 
.svg : 53 items, avg size 6K 
.ttf :2 items, avg size 421K 





Implementing a tool that reduces folder size 
by substituting duplicates with symlinks 


There are a lot of tools that compress data in various ways. The most famous 
examples for file packing algorithms/formats are ZIP and RAR. Such tools 
try to reduce the size of files by reducing internal redundancy. 

Before compressing files in archives, a very simple way to reduce disk usage 
is just deletingduplicate files. In this recipe, we will implement a little tool 
that crawls a directory recursively. While crawling, it will look for files that 
have the same content. If it finds such files, it will removeA all duplicates but 
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one. All removed files will be substituted withA symbolic links that point to 
the now unique file. This saves spaces without any compression, while at the 
same time preserving all data. 

How to do it... 

/V 

In this section, we willA implement a little tool that finds out which files in a 

directory are duplicates of each other. With that knowledge, it will remove all 

/\ 

but one of all duplicated files, and substitute them withA symbolic links, 
which reduces the folder size. 

Note 

Make sure to have a backup of your system's data. We will be 
playing with STL functions that remove files. A 

/V 

simplyA misspelled path in such a program can lead to a program 
that greedily removes too many files in unwanted ways. 

1. First, we need to include the necessary headers and then we declare that 
we use namespace std and filesystem by default. 


#include <iostream> 




#include <fstream> 

#include <unordered_map> 
#include <filesystem> 

using namespace std; 
using namespace filesystem; 


2. In order to find out which files are duplicates of each other, we will 
construct a hash map that maps from hashes of file content to the path of 

the first file from which that hash was generated. It would be a better 

/\ 

idea to use a production hash algorithm for files such asA MD5 or an 
SHA variant. In order to keep the recipe clean and simple, we just read 
the whole file into a string and then use the same hash function object 
that unordered map already uses for strings to calculate hashes. 


static size_t hash_from_path(const path &p) 

{ 

ifstream is {p.c_str(), 

ios::in | ios:ibinary}; 
if (!is) { throw errno; } 

string s; 

is.seekg(0, ios::end); 
s.reserve(is.tellg()); 
is.seekg(0, ios::beg); 

s.assign(istreambuf_iterator<char>{is}, {}) ; 

return hash<string>{}(s); 

} 

3. Then we implement the function that constructs such a hash map and 
deletes duplicates. It iterates recursively through a directory and its 
subdirectories. 


static size_t reduce_dupes(const path &dir) 

{ 

unordered_map<size_t, path> m; 
size_t count {0}; 

for (const auto Sentry : 

recursive directory iterator{dir}) { 



/V 

4. For every directory entry, it checks whether A it is a directory itself. All 

directory items are skipped. For every file, we generate its hash value 
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and try to insert it into the hash map. If the hash map A already contains 
the same hash, then this means that we already inserted a file with the 
same hash. This means that we just found a duplicate! In case of a clash 
during insertion, the second value in the pair that try empiace returns is 

false. 


const path p {entry.path()}; 

if (is_directory(p)) { continue; } 

const auto &[it, success] = 

m.try_emplace(hash_from_path(p), p); 

5. Using the return values from try empiace, we can tell the user that we 
just inserted a file because we have seen its hash for the first time. In 
case we found a duplicate, we tell the user what other file it is a 
duplicate of and delete it. After deletion, we create a symbolic link that 
replaces the duplicate. 


if ( !success) { 

cout << "Removed " << p.c_str() 

<< " because it is a duplicate of " 

<< it->second.c_str() << '\n'; 

remove(p); 

create_symlink(absolute(it->second), p); 

++count; 

} 

/\ 

6. After the filesystem iteration,A we return the number of files we deleted 
and replaced with symlinks. 


return count; 

} 

7. In the main function, we make sure that the user provided a directory on 
the command line, and that this directory exists. 



int main(int argc, char *argv[]) 

{ 

if (argc != 2) { 

cout << "Usage: " << argv[0] << " <path>\n"; 
return 1; 

} 

path dir {argv[l]}; 
if ( !exists(dir)) { 

cout << "Path " << dir << " does not exist.\n"; 
return 1; 

} 

_ /v 

8. The only thing we need to do now is to callA reduce dupes on this 
directory and print how many files it deleted. 

const size_t dupes {reduce_dupes(dir)}; 

cout << "Removed " << dupes << " duplicates.\n"; 

} 

9. Compiling and running the program on an example directory that 
contains some duplicate files looks like the following. I used the du tool 
to check the folder size before and after launching our program to 
demonstrate that the approach works. 

$ du -sh dupe_dir 
1.lMdupe_dir 

$ ./dupe_compress dupe_dir 

Removed dupe_dir/dir2/bar.jpg because it is a duplicate of 
dupe_dir/dirl/bar.jpg 

Removed dupe_dir/dir2/basel0.png because it is a duplicate 
dupe_dir/dirl/baselO.png 

Removed dupe_dir/dir2/baz.jpeg because it is a duplicate of 
dupe_dir/dirl/baz.jpeg 

Removed dupe_dir/dir2/feed_fish.jpg because it is a duplica 
dupe_dir/dirl/feed_fish.jpg 

Removed dupe_dir/dir2/foo.jpg because it is a duplicate of 
dupe_dir/dirl/foo.jpg 

Removed dupe_dir/dir2/fox.jpg because it is a duplicate of 
dupe_dir/dirl/fox.jpg 
Removed 6 duplicates. 


$ du -sh dupe_dir 



584Kdupe_dir 


How it works... 


/V 

We used the create symiinkA function in order to make a filesystem entry 
point to another file in the filesystem. This way we can avoid 
having A duplicate files. We could also have set a hard link using 
create hard iink. Semantically, this is similar, but hard links have other 
technical implications than soft links. Different filesystem formats might not 
support hard links at all, or only a certain number of hard links that refer to 
the same file. Another problem is that hard links cannot link from one 
filesystem to the other. 

However, apart from implementation details, there is one blatant error source 
when using create_symlink or create hard iink. The following lines 
contain a bug. Can you spot it immediately? 

path a {"some_dir/some_file.txt"}; 
path b {"other_dir/other_file.txt"}; 
remove(b); 

create_symlink(a, b); 

Nothing bad happens when executing this program, but the symlink will be 
broken. The symlink points to "some dir/some f ile. txt", which is wrong. 
The problem is that it should really either point to 

"/ absolute/path/some_dir/some_file.txt", or 

" . . /some_dir/some_f ile . txt". The create_symlink call uses a correct 
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absolute path if we write it A as follows: 

create_symlink(absolute(a), b); 

Note 


/V 

create symlink does not check whetherA the path we are 
linking to is correct. 



There’s more... 


We already noticed that our hash function is a too simple one. For the sake of 
keeping this recipe simple and without external dependencies, we chose this 
way. 

What is the problem with our hash function? There are actually 
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two A problems: 

• We read the whole file into a string. This is disastrous for files that are 
larger than our system memory. 

• The C++ hash function trait hash<string> is most probably not 
designed for such hashes. 

If we are looking for a better hash function, we should take one that is fast, 
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memory-friendly, and that makes sure that no twoA really large but different 
files get the same hash. The latter requirement is maybe the most important 
one. If we decide that one file is a duplicate of the other although they do not 
contain the same data, we surely have some data loss after deleting it. 

Better hash algorithms are, for example, MD5 or one of the SHA variants. In 
order to get access to such functions in our program, we could use the 
OpenSSL cryptography API, for example. 



